Decoder-based Sense Knowledge Distillation

Research Summary

Contribution Summary

• Large language models (LLMs) learn contextual embeddings that capture rich semantic information, yet they often overlook structured lexical knowledge such as word senses and relationships.
• Prior work has shown that incorporating sense dictionaries can improve knowledge distillation for encoder models, but their application to decoder as generative models remains challenging.
• In this paper, we introduce Decoder-based Sense Knowledge Distillation (DSKD), a framework that integrates lexical resources into the training of decoder-style LLMs without requiring dictionary lookup at inference time.

Why It Matters For Eval

• Extensive experiments on diverse benchmarks demonstrate that DSKD significantly enhances knowledge distillation performance for decoders, enabling generative models to inherit structured semantics while maintaining efficient training.

Related Papers

• Learning from Trials and Errors: Reflective Test-Time Planning for Embodied LLMs Embedding Neighbor
• MathFimer: Enhancing Mathematical Reasoning by Expanding Reasoning Steps through Fill-in-the-Middle Task Embedding Neighbor
• Characterizing the Predictive Impact of Modalities with Supervised Latent-Variable Modeling Embedding Neighbor
• Vichara: Appellate Judgment Prediction and Explanation for the Indian Judicial System Embedding Neighbor
• MedicalPatchNet: A Patch-Based Self-Explainable AI Architecture for Chest X-ray Classification Embedding Neighbor
• DSDR: Dual-Scale Diversity Regularization for Exploration in LLM Reasoning Embedding Neighbor
• A Survey of Data Agents: Emerging Paradigm or Overstated Hype? Embedding Neighbor
• Do We Need Adam? Surprisingly Strong and Sparse Reinforcement Learning with SGD in LLMs Embedding Neighbor
• The Statistical Signature of LLMs Embedding Neighbor
• Inner Speech as Behavior Guides: Steerable Imitation of Diverse Behaviors for Human-AI coordination Tag Overlap
• Simple Yet Effective: Extracting Private Data Across Clients in Federated Fine-Tuning of Large Language Models Tag Overlap
• EARL: Entropy-Aware RL Alignment of LLMs for Reliable RTL Code Generation Tag Overlap