看图说话终于能数清东西了
现在的AI看图说话，能说出“一只狗在草地上”，但说不出“三只狗，其中两只趴着，一只叼着红球”。要补上这些细节，过去得让模型生成、检查、重写好几轮，慢得没法用。这篇论文换了个思路：不靠多轮折腾，而是在模型开口前，直接用“图像本身的特征”当老师，逼着它把画面里的数量、材质、位置关系一次说全。结果，单次生成的精细度追平了多轮检查的效果，速度快了约20倍。它不是你明天就能用上的功能，但它指向一个趋势：AI对世界的描述，正在从“大概对”走向“数得清、说得准”。
📄 原文摘要(英文)
An image may be worth a thousand words, but most captioning models describe it in only a few. Modern vision-language models produce fluent high-level captions, yet routinely miss the attributes, counts, textures, materials, and spatial relations that make an image visually specific. Recent multi-stage systems recover some of these details through generation, decomposition, verification, and rewriting, but they do so at the expense of substantially higher inference latency. We propose SimLoss, a reference-free embedding-space objective for single-pass fine-grained image captioning. SimLoss trains a vision-language model to align its projected hidden-state representation with a frozen image embedding through an InfoNCE contrastive loss, supplying a dense visual supervision signal before any text is decoded, and requiring neither human-written fine-grained captions nor pseudo-captions from a multi-stage pipeline. We instantiate it as SimLoss FFT, which backpropagates through a locally available embedding model, and SimLoss GRPO, which treats that model as a black-box reward. Compared with single-pass, multi-stage verification, reward-optimized, and perception-aware baselines, the fully differentiable fine-tuning variant, SimLoss FFT, achieves the highest precision while nearly matching the F1 score of the multi-stage method, all while retaining single-pass inference and running roughly 20 times faster than the multi-stage pipeline. The reward-based variant SimLoss GRPO attains the strongest recall. Together, these results show that embedding-space supervision can recover the quality of multi-stage verification at the latency of a single-pass captioner.