大模型训练的秘密不在学习率，在它的影子
训练大模型时，学习率怎么调一直靠经验。这篇论文发现，真正起作用的是一个叫「有效学习率」的隐藏变量——它才是决定模型学得好不好的关键。研究者据此设计了一个新调度方案，在 1280 亿 token 的大规模训练上，比现在最常用的余弦方案损失更低、更稳。这不是你明天能直接用的技巧，但它解释了为什么有些训练技巧有效、有些无效，帮你看懂 AI 训练领域的下一步。
📄 原文摘要(英文)
Weight-normalized (WN) training is emerging as a compelling alternative to conventional weight decay (WD), with recent studies reporting substantial pretraining loss gains. We show that these gains are driven primarily by the induced effective learning rate (ELR) trajectory rather than by normalization alone. Matching ELR trajectories largely collapses the training loss dynamics of WN and WD. Schedule design should therefore target ELR rather than the nominal learning rate (LR) alone. Building on this principle, we propose AIR (Affine Inverse-square-Root decay), which combines a theoretically derived 1/t decay with an affine adjustment to satisfy prescribed peak and terminal LRs over a finite training horizon. We further identify late-stage ELR scheduling as a key source of its gains, because a shared late schedule attenuates the effect of earlier LR choices. Experiments on 128 B-token Mixture-of-Experts pretraining demonstrate that AIR outperforms cosine and other candidate schedules in training loss, downstream performance, and hyperparameter robustness, reducing terminal training loss by approximately 0.01 relative to cosine at matched cumulative LR. Applying the same ELR trajectory to WD reproduces the corresponding training dynamics, showing that the benefits extend beyond fixed-norm training and establishing ELR as the principled scheduling object.