去掉视觉编码器，大模型反而能追上？
现在几乎所有多模态大模型都先接一个预训练好的视觉编码器，相当于给模型一副“现成的眼睛”。这篇论文偏要问：如果直接让模型从原始像素学，不配这副眼睛，行不行？答案是：小规模时确实不行，但算力堆到约10^22 FLOPs(在现实预训练预算内)时，无编码器模型预计追平甚至反超。更反直觉的是，去掉编码器后，语言模型自己会长出“视觉能力”:视觉token之间的双向交互随算力增长越来越有用，视觉处理逐渐挪到更早的层，专家路由也更集中。也就是说，预训练视觉先验的优势会随规模增大而消失。它不是你明天能用上的东西，但它在暗示一个方向：多模态模型可能不需要专门的眼睛，给足算力，语言模型自己会学会看。
📄 原文摘要(英文)
Most modern multimodal large language models (MLLMs) build on a pretrained visual encoder that provides a strong visual prior. Encoder-free MLLMs instead learn visual representations directly from raw pixels, offering a simple and unified architecture, but their scaling behavior has not been systematically characterized. To fill this gap, we compare scaling laws for encoder-free and encoder-based MLLMs and report three main findings: (1) Removing the visual encoder shifts the compute-optimal allocation for the multimodal objective toward larger models, while leaving that for text nearly unchanged. (2) The two architectures exhibit nearly overlapping loss--compute frontiers on the text objective, but diverge on the multimodal objective: encoder-free models underperform at small scales yet are predicted to catch up at around 10^{22} FLOPs, well within practical pretraining budgets. (3) Without a visual encoder, the language model learns to take over its role via vision-specific adaptation: bidirectional interactions among visual tokens become increasingly beneficial as training compute grows, visual processing shifts toward earlier layers, and expert routing for visual tokens becomes more concentrated. Overall, our results indicate that the advantage of the visual prior provided by a pretrained encoder diminishes with scale, positioning encoder-free architectures as a promising direction for multimodal pretraining.