视频编辑的遥控器终于能同时按好几个键了
以前你想让 AI 改视频，要么打字说“把人去掉”，要么给一张参考图说“按这个人的样子来”，要么给一段参考视频说“镜头这么动”——一次只能用一个遥控器。这篇把三个遥控器合并成一个：文字、图片、视频参考可以同时给，数量还不限，AI 会自己分清哪条指令管内容、哪条管镜头、哪条管风格，一起作用到原视频上。它靠一个“参考银行”把各种输入统一存起来，再用一个模型同时处理原视频和这些参考，训练时还专门造了各种组合任务的数据，让模型学会多指令不打架。评测里，它在镜头、动作、内容、风格、特效五个维度上都超过了只做单一任务的专用模型和商业产品。它不是你明天就能上手用的工具，但“一次给多个参考、让 AI 自己协调”这个方向，是视频编辑 AI 从“听话”走向“懂意图”的关键一步。
📄 原文摘要(英文)
Creative intent in video editing can be expressed through text instructions (e.g., "remove the people"), image references (e.g., subject appearance), video references (e.g., camera movement), or a combination of the above. We introduce OmniXRef, a unified framework for multimodal multi-reference video editing that allows users to combine variable numbers of text, image, and video references to control diverse attributes of a source video. We first devise a unified input design with a reference-bank interface and introduce R^2-RoPE to support generalization to more references. Then our model, OmniXRef-Model, uses a single-stream diffusion transformer to jointly process the source video and multimodal context from an MLLM in an interleaved sequence. To address the scarcity of training data for video-reference editing, we construct OmniXRef-Data using synthesis pipelines for different editing tasks and combinations of references. We use curriculum learning to progressively increase task complexity and resolution. To support systematic evaluation of video as a control modality, we introduce OmniXRef-Bench, a benchmark for video-reference attribute transfer across Camera, Motion, Content, Style, and Visual Effects (VFX). Comprehensive evaluations show that our OmniXRef-Model outperforms task-specific baselines and commercial models.