一条训练数据就够？大模型蒸馏的惊人发现
训练大模型时，通常认为数据越多越好。但这项研究发现，在一种叫「在线蒸馏」的训练方式下，只用一条查询数据，模型就能持续进步数百步，并恢复大部分用完整数据集训练的效果。原因在于：模型在训练早期就快速覆盖了教师模型的大部分「状态空间」——单条查询就能触及71.5%的状态，而增加语义不同的查询到16条时，覆盖率高达98.9%，效果与全量数据持平。然而，模型吸收这些监督的速度却始终缓慢，即使状态固定也需要数百步。研究者因此提出：在线蒸馏不是「数据饥饿」，而是「算法饥饿」——数据早已喂饱，瓶颈在于模型消化速度。这挑战了「数据越多越好」的直觉，也为优化训练效率指明了新方向。
📄 原文摘要(英文)
On-policy distillation (OPD) combines student-generated rollouts with dense token-level supervision from a teacher. Existing work has mainly studied its algorithmic behavior, leaving the role of training data unclear. We examine this role at the data-minimal limit by training on a single query. One-shot OPD keeps improving for hundreds of steps and recovers most of full-data OPD's gain across task domains and model families. We explain this result through the states visited during training and the rate at which the student aligns with the teacher. We measure state coverage, the fraction of the states full-data OPD visits that a query set's rollouts reach. A single query already reaches \(71.5\%\), most of it within the first 100 steps. Adding semantically distinct queries raises coverage and validation accuracy together, until 16 queries reach \(98.9\%\) and match full-data training. Yet alignment slows at a similar pace whether OPD trains on one query or the whole dataset, and even a fixed set of states takes hundreds of steps to absorb. OPD is therefore data-overfed but algorithm-starved. Its rollouts quickly expose broad supervision, while the student absorbs that supervision increasingly slowly. The state-coverage result extends to multi-teacher OPD, where 16 semantically diverse queries per domain match full-data MOPD. As a further stress test, content-light templates and off-domain WildChat queries also approach the real-query baseline. Task content and induced state coverage can therefore come apart. We hope these findings direct future work toward the step efficiency of OPD, and prompt a re-examination of the data and the mechanisms behind its recent successes in frontier post-training.