AI Pulse
📄 论文解读

用「积木草图」实时生成可玩视频世界

现在的视频生成模型大多在「猜下一帧」,画面会慢慢漂走;这篇把「世界长什么样」和「拍成电影感画面」拆成两件事:先用粗糙的 3D 布局、甚至火柴人级别的骨架当「积木草图」,再让渲染器把它变成照片级画面。你给草图多少细节,它就守多少细节——给细的关节几何就照做,只给大方块就自己补出合理的小动作。难点在于实时:它得在四步之内出画面,还不能越跑越歪。作者用「渐进强迫」的办法,先拿真实视频的历史教模型,再让它在自己生成的画面里连续跑,30 分钟的赛车镜头里场景和车都没走样。做互动内容、虚拟拍摄、想实时「玩」一个视频世界的人,这是能摸到的那种进展;普通观众暂时还用不上。

📄 原文摘要(英文)

Visual proxies decouple explicit world-state simulation from photorealistic video synthesis, offering a controllable interface for both offline clip creation and real-time interactive streaming. Yet a practical renderer must address two challenges. Upstream proxies vary from coarse layouts to fine articulated geometry depending on the interaction, and streaming inference must operate in a few causal steps without accumulating drift across self-generated history. We present Invoker-Render, pairing a bidirectional creation model with a causal four-step streaming model under a shared conditioning interface over structural proxies, an appearance reference, and language. Trained on an aligned approximately 200-hour multi-granularity corpus, Invoker-Render honors fine geometric constraints when supplied while synthesizing plausible local motion under coarse proxies, enabling runtime selection of input detail. To bridge bidirectional creation and real-time streaming without compounding step compression and rollout drift, we introduce Progressive Forcing (ProForcing). ProForcing first trains a causal predictor and distills it into a four-step renderer using histories derived from training videos. Training then shifts to continuous self-generated rollouts. A pipelined serving stack supports efficient streaming inference. Experiments show versatile control across proxy families, granularities, and camera paths, consistent gains over recent streaming baselines, and stable 30-minute racing rollouts that preserve scene and subject appearance.

arXiv 原文

订阅 AI Pulse

每天 08:00 · 12:30 · 18:30 · 23:50 更新