AI 终于能拆开视频里的声音了
现在的 AI 生成视频，声音是一锅粥：画面里两个人对话，你只能拿到混在一起的音轨，想单独抽出某个人的声音？做不到。SepGen 把这件事拆开了：它让同一个模型在生成视频的同时，给每个画面里的声源单独出一条音轨——两个人对话，就出两条，各说各的。更妙的是它还能反向用：你丢给它一段现成的录音，再用文字描述「我要人声」「我要吉他」，它就能按描述把混音拆成独立音轨，而且拆得比专门的语音分离工具还准。这不是你明天就能用的产品，但它指向一个方向：未来的视频 AI 不再只给你「画面+声音」，而是给你「画面+可以单独编辑的声轨」，做剪辑、做混音、做沉浸式体验的人会先受益。
📄 原文摘要(英文)
A 4D audio-visual scene comprises a video, the dynamic geometry it depicts, and the sound sources that populate it, each with its own position and trajectory. Rendering such a scene from a novel viewpoint requires that every source be available as an individual waveform, so that it can be localized in the scene and propagated to the observer before the signals are mixed. Joint audio-video generators can synthesize the video and its soundtrack, but the soundtrack is emitted as a single audio-mix in which the sources are not individually accessible. We present SepGen, which extends a pretrained audio-video generator to emit the video, the mixed soundtrack, and one waveform per captioned source in one joint sampling run. SepGen supports two complementary modes: generation and separation. In generation mode, each source caption specifies what its stem contains. A two-speaker dialogue, for example, comes out as one stem per speaker in the original turn order. In separation mode, the input audio-mix remains clean while the captions specify what to extract, so the model can decompose a recording from a free-text description. We evaluate generation on scenes synthesized from text, and separation on scenes rendered by other generators and on real recordings of speech, music, and sound effects. Given an audio-mix and captions that carry the spoken lines, SepGen outperforms language-conditioned separators, most clearly on speech, and it keeps the lead when the lines are removed from the captions. Code, checkpoints, and datasets are available at https://sepgen.github.io/