开源大模型首次交出完整家底:连训练日志都给你
开源大模型以前是“开源了个模型”,训练数据、中间过程、失败记录全藏着。这次 K2-Horizon 一口气放出六个模型,从 0.9B 到 375B,更狠的是把“家底”全摊开:每个训练阶段的中间检查点、预训练数据、训练日志、全部代码,连数据是怎么造出来的流水线都写了文档。这些流水线产出了约 10 万亿 token 的合成预训练数据——合成数据第一次成了大规模预训练的主力军。技术上还有个亮点:新注意力机制让 36B 模型用七分之一参数就追平了 32B 稠密模型,解码速度还快了 2.2 倍。但对你来说,这不是明天能用的工具,而是开源社区的一次“透明化宣言”:以后谁再吹开源,可以拿这份清单当尺子量。
📄 原文摘要(英文)
We present K2-Horizon, a fleet of six fully open language models, from a 0.9B dense model to a 375B-A23B MoE, built for general-purpose reasoning and agentic work across math and science, software engineering, deep research, office automation, and beyond. The models lead or are competitive with strong open models of their size on agentic and coding benchmarks. On the modeling side, to strengthen reasoning, we introduce Mixture-of-Value Attention (MoVA), applying MoE to attention value projections, and softplus attention gating, a more flexible, theoretically motivated gate. Our 36B-A4B model uses both and nearly matches a 32B dense model trained on the same data, with one-seventh the active parameters. We also introduce Uno, lossless diffusion adapters that speed up decoding by up to 2.2x without a separate draft model. Our release follows the LLM360 definition of open source for transparency and reproducibility: the full development history, including intermediate checkpoints from every training stage, pre-training data, training logs, and all training, evaluation, and infrastructure code, notably our production-scale xLLM framework for dense and MoE training and the RL360 reinforcement-learning stack. Beyond the data itself, we release and document in detail every pipeline that built it, the part of model development that open releases most rarely disclose. These pipelines produced roughly 10T synthetic pre-training tokens, showing that synthetic data can be a major ingredient of pre-training at scale. Four of the models share one 22T-token pre-training corpus and recipe, revealing consistent scaling behavior and enabling controlled studies of scale and sparsity. All artifacts are available at https://ifm.ai/k2/