音视频AI的“注意力三角”：声音和画面会互相带偏
你让AI生成一段“雨夜咖啡馆”的视频，它可能给你配上了阳光明媚的鸟鸣——不是它笨，是它的内部机制在“串味”。研究者拆开音视频扩散模型的内部，发现文本、声音、画面三条信息流之间有一个“注意力三角”，其中声音和画面之间的通道是双向的：画面能影响声音，声音也能影响画面。当你的指令和模型从训练数据里学到的“刻板印象”冲突时，这条通道会强行把语义拉向它认为“更合理”的方向，比如把雨夜拉回晴天。这解释了为什么AI生成的多模态内容经常出现微妙的违和感。研究者还从注意力信号里提取出诊断工具，能主动诱发或控制这种串味，并在推理时做干预，让声音和画面更一致。它不是你明天能用上的功能，但如果你好奇“AI为什么总在细节上不对劲”，这就是答案。
📄 原文摘要(英文)
Audio-video diffusion models rely on cross-modal attention to coordinate text, sound, and visual content, yet this same mechanism can introduce subtle and systematic semantic leakage. We study these models by probing and analyzing the ``attention triangle,'' comprising the three cross-attention edges connecting the text, audio, and video streams, and examine how semantic information is routed across modalities during generation. Our analysis reveals that routing along the audio-video edge is bidirectional: audio can influence video generation, while video can influence audio generation. This edge is shaped by biases encoded in the model's parameters and emerges as a major contributor to leakage: when prompts are in tension with learned priors, cross-modal interactions may override the intended conditioning and reroute semantics toward visually canonical but incorrect outcomes. These effects suggest that semantic artifacts arise not merely from attention spreading beyond its intended target, but from structured, bias-driven interactions along specific pathways. Building on this perspective, we extract attention-derived signals that expose how semantics are distributed and grounded across modalities, and use them as a diagnostic tool to both analyze and deliberately incur leakage under controlled conditions. This enables us to probe the internal dynamics of cross-modal routing and isolate the role of individual interactions. We further leverage these signals to guide inference-time interventions that encourage more consistent cross-modal alignment. Extensive experiments support our analysis and demonstrate improved semantic grounding while preserving generation quality.