大模型看图的成本，终于能按内容省了
多模态大模型看一张图，不管内容多简单，都要把整张图切成固定大小的方块，每个方块都按同样的精度处理——这就像把一张白纸和一张密密麻麻的报表用同样的分辨率扫描。这篇论文让模型自己决定哪里看仔细、哪里扫一眼：它先粗粗地看全图，再让一个“路由器”判断哪些区域值得放大，哪些可以保持模糊，训练时只靠模型自己教自己，不依赖人工标注。在27B参数的模型上，平均省下43%的视觉计算量，性能只掉1.1%，而传统的固定砍半方法性能掉12%。部署到推理引擎后，吞吐量提升2.3倍，首字延迟降54%。它不是你明天能用上的功能，但“按需分配注意力”是下一代多模态模型绕不开的方向。
📄 原文摘要(英文)
Multimodal large language models have become the dominant paradigm for visual understanding, but incur substantial costs by encoding inputs into dense, fixed-size patch tokens. However, visual information is unevenly distributed: some regions require fine-grained detail, while others admit compact representations. Downsampling sacrifices this detail, while existing token pruning and adaptive approaches remain limited in content-adaptive granularity, task generalization, and integration with modern MLLMs and serving infrastructure. Overcoming these limitations calls for foundation models that learn, end to end, where-and at what granularity-to allocate visual representations, a native capability we term elastic visual representation weaving. We introduce VisionWeave, establishing this capability in frontier-level MLLMs through large-scale training. It combines two components: a gated spatial pooler constructs coarse-grained representations alongside native fine-grained representations within a shared MRoPE coordinate, while a granularity router learns their content-adaptive allocation. Through self-distillation alone, we validate this capability on Qwen3.5-4B and scale to Qwen3.8-27B with over 30K A100 GPU-hours. Based on Qwen3.8-27B, VisionWeave adaptively adjusts token savings to visual content, saving 43.0% tokens on average while retaining 98.9% native performance across eight benchmarks, versus only 88% performance preserved for token pruning baselines with a fixed 50% savings target. Extensive evaluations confirm robust efficiency-quality trade-offs across diverse tasks, resolutions and video frames. When deployed on SGLang serving engine, our method achieves a 2.3x throughput gain while reducing mean TTFT by 54.4% and mean TPOT by 60.6%. Together, we believe these results position elastic visual weaving as a promising capability for next-generation multimodal models.