GATES: Self-Distillation under Privileged Context with Consensus Gating

Research Summary

Contribution Summary

• We study self-distillation in settings where supervision is unreliable: there are no ground truth labels, verifiable rewards, or external graders to evaluate answers.
• We focus on document-grounded question answering with asymmetric context, where a single model serves as both tutor (with access to a relevant source document during training) and student (answering from the question alone at test time).
• Rather than assuming tutor correctness, we derive supervision online from tutor consensus by sampling multiple document-grounded reasoning traces and using agreement to gate learning.

Why It Matters For Eval

• Held-out in-domain accuracy under asymmetric evaluation improves from 46.0\% to 62.0\%, and average (maj@8) accuracy on public document-free math benchmarks improves from 20.2\% to 35.4\%.

Related Papers

• OpaqueToolsBench: Learning Nuances of Tool Behavior Through Interaction Embedding Neighbor
• Can Multimodal LLMs Perform Time Series Anomaly Detection? Embedding Neighbor
• Actor-Curator: Co-adaptive Curriculum Learning via Policy-Improvement Bandits for RL Post-Training Embedding Neighbor
• Multimodal Integrated Knowledge Transfer to Large Language Models through Preference Optimization with Biomedical Applications Embedding Neighbor
• Pretraining Language Models for Diachronic Linguistic Change Discovery Embedding Neighbor
• KLong: Training LLM Agent for Extremely Long-horizon Tasks Embedding Neighbor
• Continuous Telemonitoring of Heart Failure using Personalised Speech Dynamics Embedding Neighbor
• Towards Expectation Detection in Language: A Case Study on Treatment Expectations in Reddit Embedding Neighbor
• LiLo-VLA: Compositional Long-Horizon Manipulation via Linked Object-Centric Policies Embedding Neighbor
• Watermarking LLM Agent Trajectories Tag Overlap
• MathScape: Benchmarking Multimodal Large Language Models in Real-World Mathematical Contexts Tag Overlap
• Unlocking Reasoning Capability on Machine Translation in Large Language Models Tag Overlap