AI终于能生成音画同步的视频了
现在的AI生成视频大多是默片，声音和画面是分开训练的，导致两者对不上。这篇论文让AI同时学音频和视频，用对比学习把声音和画面在时间上对齐——比如敲键盘的声音和手指动作匹配。实验证明，对齐后的视频生成质量更高，音画同步更准。它不是你明天能用上的，但意味着未来AI生成的视频不再像配音版PPT。
📄 原文摘要(英文)
Recent generative models are moving beyond silent video or standalone audio synthesis toward the joint generation of synchronized audio and video. Despite this progress, jointly generating audio and video with fine-grained cross-modal correspondence remains challenging due to their fundamental structural differences. Most existing methods use audio and video VAEs trained separately. As a result, the two latent spaces lack cross-modal alignment, leaving the downstream generative model to learn cross-modal synchronization from scratch. We present OmniVAE, a jointly trained audio-video VAE that learns fine-grained semantic alignment between audio and video latent representations. Beyond reconstruction, OmniVAE uses a segment-level audio-video contrastive objective to capture temporal-semantic correspondence and align the two latent spaces. In parallel, it distills features from pretrained modality-specific semantic encoders into each modality, improving the downstream learnability of both latent spaces. Extensive experiments show that both objectives consistently improve the learnability of the latent spaces, translating into higher generation quality and more accurate cross-modal synchronization in downstream text-to-audio-video generation. These findings underscore the importance of learning unified representations as a foundation for omnimodal modeling.1