图像分词器不只是压缩图片，它悄悄改写了模型学语言的方式
图像分词器——把图片切成模型能读的“视觉单词”的模块——一直被当成独立零件来评测：重建得像不像、生成好不好。这篇论文换了个角度：把它放进和文本一起训练的真实环境里，盯着训练过程中各项任务的损失曲线。结果发现三件反直觉的事：重建得越像原图，不代表下游任务做得越好；分词器的选择甚至会反过来影响模型学文本的能力；同一个分词器，在不同任务上的表现排名会完全颠倒。换句话说，视觉语言不是孤立存在的，它和文本语言在训练中互相拉扯。研究者用这个测试平台重新审视了三个设计选择(判别器、语义监督、词表大小)，发现它们对联合建模的影响和单独评测时的结论并不一致。这不是你明天能直接用的技术，但它提醒我们：AI 的“视觉理解”和“视觉生成”可能共享同一套底层语言，而我们对这套语言的理解还停留在零件层面。
📄 原文摘要(英文)
Image tokenizers define the ``visual language'' of unified multimodal models, yet are commonly studied through isolated metrics or generation-/understanding-only evaluations. These evaluations do not fully capture how visual tokens behave when modeled jointly with text. We build a controlled pure-autoregressive testbed and track task-specific validation losses during multimodal continual pretraining across text, image, text-to-image (T2I), and image-to-text (I2T) prediction. We examine how these losses scale and relate to downstream performance, then use them to study multimodal learnability---how well image and text tokens are jointly modeled---and tokenizer design. We find that (1) losses should be analyzed by task, since they exhibit distinct scaling behavior and rank tokenizers differently. (2) The loss--performance relationship depends on the predicted token space: for a fixed tokenizer, T2I and I2T losses correlate with generation quality, but across tokenizers, the T2I loss--performance relationship shifts with the image-token space, whereas I2T loss, computed over a shared text vocabulary, provides a more consistent signal. I2T loss also correlates with both generation and visual understanding performance after supervised finetuning. Using losses as a lens, we show that (3) better reconstruction does not necessarily yield lower task-specific losses or stronger downstream performance, and that (4) image tokenizer choice can affect text modeling under joint optimization. As case studies, we revisit three tokenizer design axes---the discriminator, semantic supervision, and vocabulary size---to examine their effects on joint modeling and downstream performance. Together, our testbed offers a complementary perspective on image tokenizers as visual languages, highlighting their interplay with text in joint multimodal training.