大模型看图，不需要预训练视觉编码器
多模态大模型怎么「看」图，一直是个黑箱。主流做法是拿一个预训练好的视觉编码器当眼睛，但新研究用严格对照实验拆开三个变量，得出反直觉结论：预训练这个环节根本不必要，从零和语言模型一起训练的编码器效果一样好。真正关键的是「专用视觉容量」——模型得有专门处理图像的参数，哪怕这些参数不放在最前面、而是融进主干网络早期层，效果也不输传统方案，还能加快首字输出。这动摇了「预训练视觉编码器不可替代」的行业默认，对做多模态模型的人是个明确信号：别再迷信预训练，把视觉能力交给联合训练。
📄 原文摘要(英文)
How multimodal language models should encode visual input remains up for debate. Most models use a pretrained vision encoder, while some recent recipes train the encoder from scratch with the language model or remove it entirely. Vision encoders bundle together three design choices: a visual prior from upstream pretraining, dedicated vision capacity, and pre-fusion placement of that capacity (i.e., before the multimodal backbone). Because published models also differ in data, architecture, and compute, comparing them cannot isolate the effect of each choice. We disentangle these choices via a controlled framework where we pretrain all models from scratch on common data and compare them on visual question answering and language benchmarks at matched lifetime compute. We find that the visual prior is not necessary; an encoder trained jointly from scratch is on par with a pretrained one, without a separate training stage. What matters instead is dedicated vision capacity: an encoder-free model trails on visual understanding even when its backbone is enlarged by the encoder's parameter count. Yet that capacity need not precede multimodal fusion, as vision-only modules added in-fusion to early backbone layers perform comparably to a pre-fusion encoder and can lower time to first token. These results also hold across backbone sizes, when encoder pretraining is treated as free, and when starting from a pretrained language model. Overall, our findings favor dedicated vision capacity trained jointly with the backbone and placed early within it.