大模型学会先想概念再说话，训练省一半
大模型现在学说话，是一个字一个字往外蹦。这篇让它先学会「想概念」:把多个字打包成一个概念，先预测接下来要讲什么概念，再回到字级别把话说完。8.9B 参数的模型只用了 51.3% 的训练数据，就达到了同规模开源模型的最终水平；下游任务平均分反超 2.45 分，数学题 GSM8K 上多拿 5.99 分。更妙的是，训练完这个概念层还能单独拿来用：只更新 1700 万参数的小模块，就能让模型快速适应新领域；把概念表示塞给推理加速器，平均多接受 4.17% 的生成步数，几乎零开销。它不是你明天能直接调用的 API，但「先想后说」这条路，可能是大模型省算力的下一个方向。
📄 原文摘要(英文)
We introduce NCP-ArchPreview, a latent-space language model that pushes autoregressive pretraining beyond standard next-token prediction (NTP). Alongside NTP, the model learns through Next Concept Prediction (NCP) to predict discrete concepts that span multiple tokens, introducing an explicit and more challenging concept-level objective while preserving standard token-level autoregressive generation. NCP-ArchPreview builds a latent space by constructing a product-quantized concept vocabulary directly from its hidden states, and subsequently learns to predict future concepts via a dedicated Concept Module. These predicted concepts are then fed back to the token level to guide subsequent generation, with NTP and NCP trained jointly end-to-end. We scale this architecture to 8.9B parameters and train it on 5.73T tokens from the Dolma-3 dataset, marking the largest demonstration of a latent-space language model to date. Remarkably, by consuming only 51.3% of the total training tokens, NCP-ArchPreview achieves the final pretraining loss of OLMo-3-7B. Following full pretraining, it outperforms OLMo-3-7B by 2.45 points on the downstream macro-average, including a notable 5.99-point gain on GSM8K. Controlled experiments isolate a clear progression of performance gains stemming from both the latent architecture and the NCP objective. Furthermore, utilizing only 85% of the standard computation, NCP-ArchPreview approaches the training loss of a strictly parameter-aligned 8.9B baseline. The learned latent space remains highly valuable after the pretraining stage: updating just the 17M-parameter VQ module yields a novel, lightweight interface for domain adaptation, while a simple injection of concept representations into a DFlash2 drafter improves the mean accepted length by 4.17% with negligible overhead.