2B模型训练成本从150万美元降到6900美元
训练一个像样的语言模型，过去是百万美元级别的游戏：复现 Llama-3.2-3B 要花 150 万美元，复现 SmolLM3-3B 也要 70 万美元。这篇把门槛砸到了 6900 美元——用一块消费级 RTX 5090 显卡，FP8 低精度训练，从零训出一个 20 亿参数的模型，性能逼近 Qwen2.5-1.5B。省钱的招数不是单点突破，而是组合拳：选对硬件、低精度、超球优化、课程模型平均，外加数据配方。他们还顺手总结出一条「成本缩放定律」:按这个趋势，花 4400 美元就能追平 Qwen2-1.5B。这不是你明天能用的工具，但它是「大模型训练平民化」的一个真实坐标：当训练成本从百万级掉到几千美元，高校实验室和个人研究者第一次有了上桌的可能。
📄 原文摘要(英文)
Language model pretraining has become almost synonymous with prohibitive cost, placing it out of reach for much of the academic and open-source communities. Although strong open-source efforts already exist, including open-weight models and open-source training recipes, a cost-efficient, hardware-accessible, and open-source pretraining recipe has long been missing. Even at a small scale, training Llama-3.2-3B costs over \1.5M, and reproducing SmolLM3-3B needs over 700K. In this report, we present an open pretraining recipe designed to lower this barrier. Using this recipe, we train a collection of Puro-2B models from scratch on up to 1.4 trillion tokens with FP8 precision on consumer-grade RTX 5090 GPUs. The models in the collection differ in token budgets and selected recipe variants. Our best model is trained at a compute cost of less than \6.9K and approaches Qwen2.5-1.5B performance under our evaluation protocol. This cost efficiency is enabled by a combination of approaches, including hardware selection, low-precision training, hyperball optimization, curriculum model averaging, and the data recipe. Beyond the recipe itself, we provide two additional results. First, across the Puro-2B collection, we derive a Puro Cost Scaling Law that relates training cost to average model performance; the fitted law suggests that about 4.4K, less than \$5,090, is sufficient to reach the performance of Qwen2-1.5B. Second, as an end-to-end case study, we examine how pretraining data curricula shape downstream performance after post-training. Such controlled studies are enabled by having access to the full pretraining pipeline rather than model weights alone. We release the full training recipe for Puro-2B, including data, code, and model weights under Apache 2.0 at https://huggingface.co/collections/thu-pacman/puro-2b.