AI 看图说话卡壳的那几步,被盯上了
多模态扩散模型生成答案是一步步「去噪」磨出来的,研究者发现:整体上模型越磨越确定,但中间有几步会突然「慌」一下——局部不确定性骤升,而这几步恰好和它胡编内容高度重合。他们的做法是不在这几步硬把候选词定死,而是保留多个候选的加权混合、多磨几轮再定,等分布稳了再切回正常流程。这是个即插即用的解码策略,不用重训模型,在多个多模态基准上都能压住幻觉。它不是你明天能直接上手的东西,但如果你在选多模态模型做产品,这类「推理时干预」意味着幻觉可以在不改模型权重的前提下被削一层。
📄 原文摘要(英文)
Recent advancements in multimodal diffusion language models (MDLMs) have exhibited strong global modeling capabilities through iterative refinement. We observe that mask uncertainty follows an overall downward trend during iterative denoising, while a few denoising steps exhibit localized entropy spikes closely associated with hallucinations. We argue that anomalous steps can be identified by measuring the entropy deviation of each denoising step from its neighboring temporal window. We hypothesize that discrete token commitment discards the distributional information at masked positions, limiting the continued interaction between candidate semantics and the context during entropy spikes. With this goal, we present StepRefine, a plug-and-play decoding strategy that leverages semantic context to achieve reliable denoising. The core of our method lies in a step-aware switch between discrete denoising and latent refinement. The model employs probability-weighted embeddings at entropy spike steps to preserve diverse reasoning hypotheses and performs multiple rounds of continuous refinement. Meanwhile, a refinement controller monitors distributional stability to reduce unnecessary loops and switch the model back to standard discrete denoising. Through extensive experiments, StepRefine demonstrates significant hallucination mitigation across different MDLMs on multimodal benchmarks.