Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?
Jeonghye Kim, Xufang Luo, Minbeom Kim, Sangmook Lee, Dohyung Kim +3 more
Abstract
Self-distillation has emerged as an effective post-training paradigm for LLMs, often improving performance while shortening reasoning traces. However, in mathematical reasoning, we find that it can reduce response length while degrading performance. We trace this degradation to the suppression of epistemic verbalization - the model's expression of uncertainty during reasoning. Through controlled experiments varying c...
Best Implementation
Maintained implementation evidence is not confirmed for this paper yet.
Use the Implementation Status and Reproduction Path sections below for the current action plan.
Reproduction Path
Follow this baseline workflow to decide if this paper is worth immediate prototyping.
- 1
This is primarily a method paper. Reproduce it within a maintained framework baseline instead of chasing paper-specific repos.
- 2
Start with framework-native implementations (e.g. PyTorch optimizer module, Optax, or Transformers training loops).
- 3
Replicate the paper ablation settings first, then compare against modern baselines.
Additional Implementations
Official
No additional official repositories detected.
Community
- beanie00/self-distillation-analysisConfidence: low
Codebase for the work “Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?”
Stars: 46Forks: 6Last push: Mar 2026License: Apache-2.0
Hugging Face Artifacts
No trustworthy direct or curated related Hugging Face artifacts were found yet.
Continue with targeted Hugging Face searches: