多模态预训练的秘密：数据复杂度决定AI是协作还是内耗
多模态AI训练有个反直觉的发现：数据复杂度决定了不同模态是互相帮忙还是互相拖累。研究者用受控实验拆解了语言、视觉理解、视觉生成之间的知识流动，发现它们的影响是不对称的——有的模态是主导，有的只是陪跑。更关键的是，架构选择能改变这种关系：共享注意力加模态专属的前馈层能促进协同，而延迟整合会让模型变懒，只靠语言先验糊弄视觉任务。他们还给出了一个实用配方：只用5%的计算量就能达到强生成性能。这不是你明天能直接用的工具，但它解释了为什么有些多模态模型越训越聪明，有些却原地踏步——对想理解AI能力边界的人来说，这是值得记住的一课。
📄 原文摘要(英文)
Vision offers a critical axis for advancing foundation models, driving a shift towards natively unified multimodal pretraining. Despite this momentum, the design space and the fundamental mechanisms of how modalities interact during unified training remain underexplored. We provide empirical clarity through a systematic exploration of multimodal pretraining. Our controlled experiments on both synthetic and large-scale real-world datasets yield four key insights into the physics of multimodal pretraining: (i) Knowledge Flow: We disentangle how language, visual understanding, and visual generation transfer knowledge across modalities, revealing distinct patterns of influence and asymmetry; (ii) Synergy vs. Competition: We show that data "complexity" largely determines whether modalities are synergistic, identify architectural choices that promote synergy: such as shared attention and normalization with modality-specific feed-forward layers, and find that these behaviors generalize across different visual tokenizer designs; (iii) Early Unification: Unifying modalities from the very early stages and training them jointly is shown to be more effective than late alignment or sequential training. This process uncovers a vision laziness phenomenon, where delayed integration leads models to rely on language priors; (iv) Recipes: We derive efficient pretraining recipes that achieve strong generative performance using only 5% of the compute budget. These core findings are subsequently validated at scale by training multiple 13.5B MoE models on 2T tokens. We hope this study provides a principled foundation for understanding and scaling multimodal pretraining.