Error-Aware Knowledge Distillation via Targeted Revision for Customer-Service Summarization

Research Summary

Contribution Summary

• We introduce an Analyze-Revise-Finetune (ARF) pipeline that enables smaller open-source language models (LLMs) to surpass substantially larger proprietary models in customer service summarization tasks.
• The pipeline first analyzes and categorizes common errors in summaries produced by a teacher model (GPT-3.5), then performs a targeted revision using a compact editor model (Llama 3.1 70B) to generate high-quality, refined training data.
• Fine-tuning smaller student models (e.g., Llama 3.1 8B, QWen3 4B) on this refined data resulted in superior summarization performance compared to GPT-3.5.

Related Papers

• BETA-Labeling for Multilingual Dataset Construction in Low-Resource IR Embedding Neighbor
• When LLMs Imagine People: A Human-Centered Persona Brainstorm Audit for Bias and Fairness in Creative Applications Embedding Neighbor
• A Survey on the Optimization of Large Language Model-based Agents Embedding Neighbor
• Beyond Static Pipelines: Learning Dynamic Workflows for Text-to-SQL Embedding Neighbor
• CricBench: A Multilingual Benchmark for Evaluating LLMs in Cricket Analytics Embedding Neighbor
• Beyond the Star Rating: A Scalable Framework for Aspect-Based Sentiment Analysis Using LLMs and Text Classification Embedding Neighbor
• Manipulating language models' training data to study syntactic constraint learning: the case of English passivization Embedding Neighbor
• One Brain, Omni Modalities: Towards Unified Non-Invasive Brain Decoding with Large Language Models Embedding Neighbor
• Language Statistics and False Belief Reasoning: Evidence from 41 Open-Weight LMs Embedding Neighbor
• SocialHarmBench: Revealing LLM Vulnerabilities to Socially Harmful Requests Tag Overlap
• ClearFairy: Capturing Creative Workflows through Decision Structuring, In-Situ Questioning, and Rationale Inference Tag Overlap
• Beyond Refusal: Probing the Limits of Agentic Self-Correction for Semantic Sensitive Information Tag Overlap