A Proof of Learning Rate Transfer under $μ$P

Research Summary

Contribution Summary

• We provide the first proof of learning rate transfer with width in a linear multi-layer perceptron (MLP) parametrized with $μ$P, a neural network parameterization designed to ``maximize'' feature learning in the infinite-width limit.
• We show that under $μP$, the optimal learning rate converges to a \emph{non-zero constant} as width goes to infinity, providing a theoretical explanation to learning rate transfer.
• In contrast, we show that this property fails to hold under alternative parametrizations such as Standard Parametrization (SP) and Neural Tangent Parametrization (NTP).

Related Papers

• TAROT: Test-driven and Capability-adaptive Curriculum Reinforcement Fine-tuning for Code Generation with Large Language Models Embedding Neighbor
• Compose and Fuse: Revisiting the Foundational Bottlenecks in Multimodal Reasoning Embedding Neighbor
• Evidence-Grounded Subspecialty Reasoning: Evaluating a Curated Clinical Intelligence Layer on the 2025 Endocrinology Board-Style Examination Embedding Neighbor
• Understanding Artificial Theory of Mind: Perturbed Tasks and Reasoning in Large Language Models Embedding Neighbor
• Diagnosing LLM Reranker Behavior Under Fixed Evidence Pools Embedding Neighbor
• Learning User Interests via Reasoning and Distillation for Cross-Domain News Recommendation Embedding Neighbor
• SPELL: Self-Play Reinforcement Learning for Evolving Long-Context Language Models Embedding Neighbor
• Latent-Augmented Discrete Diffusion Models Embedding Neighbor
• MolReasoner: Toward Effective and Interpretable Reasoning for Molecular LLMs Embedding Neighbor
• GATES: Self-Distillation under Privileged Context with Consensus Gating Tag Overlap
• Proof-RM: A Scalable and Generalizable Reward Model for Math Proof Tag Overlap
• Esoteric Language Models: Bridging Autoregressive and Masked Diffusion LLMs Tag Overlap