AI 终于能同时生成画面和声音了
以前的 AI 视频生成器要么没声音，要么声音是后配的，画面和声音各干各的。这篇把两者真正焊在一起：一个 7B 参数的模型，输入一张首帧和一句文字，直接同时生成同步的 2K 视频和音频。它靠的是在模型后半段加了一个「门控交叉注意力」，让画面和声音在生成过程中互相参考，而不是各走各路。训练上还用了强化学习，专门把视频、音频、以及两者是否匹配的反馈分开喂给模型。结果在开源系统里达到顶尖水平，而且模型和 2K 增强器都开源了。它不是你明天就能用的工具，但这是 AI 生成从「哑巴视频」走向「有声电影」的关键一步。
📄 原文摘要(英文)
Recent video generators often omit audio or synthesize it in a separate stage, limiting reciprocal modeling of visual dynamics and acoustic events. We present DreamX-Creator 1.0, a compact native joint audio-video generation system centered on a 7B generator. Conditioned on a first frame and a text prompt, the generator jointly denoises modality-specialized audio and video streams. The streams are processed independently in the first half of the network and coupled in the latter half through Gated Cross-Modal Attention, whose token- and head-wise output gates modulate each active cross-modal attention-head output. A unified Audio-Video Data System constructs and filters temporally coherent clips, produces structured multimodal annotations, and organizes clips into capability-oriented data pools. Progressive Joint Training comprises two audio-video pre-training stages followed by High-Quality Finetuning. Audio-Video Reinforcement Learning further post-trains the generator with Modality-Aware Multimodal Feedback that routes video-, audio-, and cross-modal feedback to the corresponding streams. For high-resolution output, our Autoregressive 1-Step 2K Refinement pipeline adapts a bidirectional multi-step teacher into an autoregressive multi-step refiner and distills it into a student requiring one denoising evaluation per temporal chunk. Overall, DreamX-Creator 1.0 achieves native, synchronized audio-video generation with performance competitive with state-of-the-art open-source systems. By releasing our compact 7B generator and 2K Refiner, we seek to democratize native audio-video generation and provide an accessible foundation for future research in unified audio-video generative modeling.