被抛弃的层丢弃法，反而能省25%训练算力
大模型训练圈一度把「层丢弃」当成过时技巧丢掉，因为有人报告它会掉精度。这篇用2400多次实验、最大82亿参数的规模证明：不是方法不行，是没人认真调过它。把丢弃的层分布、时间表、优化器参数一起调好，同样的训练算力下损失更低；固定训练步数时，能省最多25%的算力，验证集损失不升反降。更妙的是，训练时用层丢弃，推理时就能让模型提前退出、跳层、自我投机解码，速度最多快1.5倍，精度几乎不掉。它不是你明天就能抄的配置——具体怎么调还得看论文——但它说明：大模型社区里被主流抛弃的旧招，可能只是缺一次认真对待。
📄 原文摘要(英文)
Layer dropout (a.k.a. stochastic depth) has been shown to enable faster training, higher accuracy, and robustness to zero-shot layer pruning in both language and vision transformers. However, as models and datasets have scaled, dropout - particularly layer dropout - has largely disappeared from large language models (LLMs) pre-training recipes. While some prior work has reported that dropout can degrade accuracy, no comprehensive study has quantified, let alone mitigated, this effect. In this study, we show that layer dropout should be used in state-of-the-art LLM training, establishing best practices and scaling analysis for both training and post-training benefits. Concretely, with optimal layer distribution, time schedule, and optimizer hyperparameters, we observe that at the same training FLOPs layer dropout leads to lower loss. For a given number of training steps, LLMs can achieve lower or similar validation loss while saving upto 25% of training FLOPs. Moreover, layer dropout enables significant post-training optimizations, such as early exit, intermediate-layer skipping, and self-speculative decoding, yielding up to 1.5x inference speedup with negligible accuracy loss. Across more than 2400 training experiments, spanning models from 271M to 8.2B parameters and datasets up to 160B tokens, we demonstrate that these findings extend reliably to large-scale training regimes. All pre-training experiments were run on Cerebras CS-3 systems.