大模型训练四阶段，被一个算法统一了
大模型训练一直分四步走：预训练、中训练、微调、强化学习，每步换数据也换算法。这篇论文说，这四步其实可以是一个算法：把强化学习的目标改写成「最大似然拟合模型自己生成的数据」，于是整个训练流程只剩一个问题——喂什么数据、按什么比例、什么时候喂。传统分阶段训练只是这个统一框架的特例。他们在同一个训练器里混着静态数据和强化学习生成的数据跑，结果在同样的算力下，准确率和任务覆盖都更好，还保留了原有能力，给后续持续学习留了更好的起点。它不是你明天能用上的东西，但它把「训练大模型」从一门手艺变成了一门可以推导的科学。
📄 原文摘要(英文)
Foundation models are typically trained in stages: pretraining, midtraining, supervised fine-tuning (SFT), and reinforcement learning (RL), with changes in both data and learning algorithms. We show that this algorithmic separation is not fundamental. By deriving suitable RL objectives as maximum-likelihood fitting to model-generated target distributions, we place all of these stages under a single likelihood-based training algorithm, unified training. This perspective turns the foundation-model training lifecycle into a problem of choosing which data to train on, in what proportions, and when, with conventional staged training as a special case. We instantiate this framework in a unified training system that mixes static data and RL generations within a single trainer. Across controlled and real-world experiments, unified training improves the accuracy-compute tradeoff and task coverage while preserving broader capabilities, and provides a stronger starting point for continual learning. This work lays the groundwork for a unified science of foundation-model training, connecting mathematical principles, training infrastructure, and the development of model capabilities.