多模态预训练的秘密：早融合比晚对齐强
多模态模型不是把语言和视觉数据混在一起训练就行。这篇论文用受控实验拆开了多模态预训练的“物理”：语言、视觉理解、视觉生成之间如何互相传递知识，发现数据复杂度决定模态是互相促进还是互相打架；架构上共享注意力、但各模态用独立的FFN层能促进协同；最关键的是，从最早阶段就统一训练多模态，比先分开训再对齐或顺序训练更有效——晚融合会让模型偷懒，只靠语言先验，出现“视觉懒惰”。基于这些规律，他们用5%的计算量就训出了强生成性能的模型，并在13.5B MoE模型上验证。这不是你明天能直接用的技巧，但它解释了为什么多模态模型有的聪明有的笨，给训练这类模型的人指了条更省算力的路。
📄 原文摘要(英文)
Vision offers a critical axis for advancing foundation models, driving a shift towards natively unified multimodal pretraining. Despite this momentum, the design space and the fundamental mechanisms of how modalities interact during unified training remain underexplored. We provide empirical clarity through a systematic exploration of multimodal pretraining. Our controlled experiments on both synthetic and large-scale real-world datasets yield four key insights into the physics of multimodal pretraining: (i) Knowledge Flow: We disentangle how language, visual understanding, and visual generation transfer knowledge across modalities, revealing distinct patterns of influence and asymmetry; (ii) Synergy vs. Competition: We show that data "complexity" largely determines whether modalities are synergistic, identify architectural choices that promote synergy: such as shared attention and normalization with modality-specific feed-forward layers, and find that these behaviors generalize across different visual tokenizer designs; (iii) Early Unification: Unifying modalities from the very early stages and training them jointly is shown to be more effective than late alignment or sequential training. This process uncovers a vision laziness phenomenon, where delayed integration leads models to rely on language priors; (iv) Recipes: We derive efficient pretraining recipes that achieve strong generative performance using only 5% of the compute budget. These core findings are subsequently validated at scale by training multiple 13.5B MoE models on 2T tokens. We hope this study provides a principled foundation for understanding and scaling multimodal pretraining.