视频重拍从几分钟缩到几秒,相机还听你指挥
以前想让 AI 沿着新相机轨迹重拍一段视频,得等它一步步慢慢算,几分钟起步;这篇把采样步数砍到原来的 1/25,几秒出片,画质和几何一致性还追平了多步模型。关键不是硬加速,而是先修好一个隐蔽的毛病:现有模型在不同采样步数下对相机控制的理解不一致,导致轨迹弯弯绕绕、难收敛。研究者用视觉几何模型把源视频的隐藏特征和目标视角对齐,让几何变换直接编码进视频流,轨迹变直了,再配合新的训练目标和蒸馏策略,把误差一步步压掉。它不是你明天就能用的工具,但这是生成式渲染从实验室走向实时工具的关键一步。
📄 原文摘要(英文)
We present FlashRender, a few-step generative rendering framework that retakes a source video along a target camera trajectory in seconds. We identify sampling-step-dependent camera control as a prominent manifestation of discretization error in existing multi-step generative rendering models and show that resolving this inconsistency substantially lowers denoising trajectory curvature, facilitating subsequent step distillation. To this end, we introduce Representation Transformation and Alignment (RETA), which aligns hidden source-video representations with target-video features from a frozen visual geometry model. This directly encodes the geometric transformation within the source-video stream, enabling sampling-step-consistent camera control. We then fine-tune the model with the MeanFlow objective on the lower-curvature denoising trajectory induced by RETA, allowing the model to more effectively address discretization error. Finally, we apply on-policy flow map distillation to correct self-rollout errors under fixed few-step sampling. Extensive experiments show that RETA, MeanFlow, and on-policy flow map distillation play complementary roles in few-step generative rendering. Together, they enable our approach to match multi-step baselines in video quality and geometric consistency at 25x lower sampling cost while achieving superior camera controllability, even under out-of-distribution target camera trajectories.