换脸换声终于能同步了：一个模型搞定视频里的脸和声音
以前的AI换脸视频，脸和声音是分开处理的：一个模型换脸，另一个模型换声，结果经常对不上——嘴型动了声音没跟上，或者声音变了脸还是原样。UniSwap是第一个把这两件事放进同一个模型里做的框架：你给一段视频、一张参考照片、一段参考声音，它一次性把脸和音色都换掉，同时保留原视频的动作、场景和说话内容。它靠一个“先抹掉再重建”的训练技巧解决了配对数据稀缺的问题，还通过一系列工程优化把生成速度从30步降到3步，支持实时流式输出。这不是你明天就能用的工具，但它意味着“数字人换脸换声”从各管各的拼装，走向了真正同步的一体化，是视频合成领域一个值得关注的方向。
📄 原文摘要(英文)
Talking-video character replacement requires coordinated transfer of appearance and voice while preserving the source motion, scene, linguistic content, and audio-video timing. Existing methods use separately optimized models for the two modalities, making audio-visual consistency difficult to enforce. We present UniSwap, the first framework for streaming joint audio-visual identity replacement in talking videos. Given a source video, a reference image, and a reference voice clip, UniSwap transfers the reference appearance and vocal timbre within a single audio-visual diffusion transformer while preserving the source content and dynamics. To address the scarcity of aligned cross-identity training pairs, we introduce a swap-and-reconstruct pipeline that removes visual and vocal identity from real clips and uses the original clips as reconstruction targets. Starting from a bidirectional backbone, we progressively adapt the model through In-context Pretraining for joint replacement, Conditional Streaming Adaptation for block-causal KV-cached generation, and Efficient Self-forcing DMD for mitigating exposure bias and reducing sampling from 30 to 3 denoising steps per block. Efficient Multi-LoRA Switching enables the three DMD roles to share a single frozen backbone. Feature-RoPE Decomposition keeps cached positions within the training range, supporting stable long-form inference. Experiments demonstrate strong audio-visual synchronization, competitive identity preservation, efficient streaming, and stable long-form generation.