用手直接捏出视频世界
现在的视频世界模型,你只能用文字和镜头方向去「逛」,想动画面里的东西,没门。这篇把手变成了遥控器:你手怎么动,画面里的物体就怎么被推、被拿、被摆。难点在于手和物体的互动太细碎,还要实时算。他们先攒了 10 万段手部交互片段,覆盖真实和合成场景,专门给手部动作和物体接触做监督;再把生成过程压成几步,用一套把视频先验和图像先验分开打分的办法,让互动动作和画面质感各自被修好;最后换了个更轻的视频解码器,保住画质的同时把算力砍下来。结果是在单张 H200 上,540p 画面能跑到每秒 18 帧、延迟 741 毫秒,手部动作的准确度(MANO Dice)在实时三步模型上还有 92.49。它离你明天能用上还差一步——得先有这套模型跑在你手边——但它指向的东西很具体:以后你对着屏幕比划,画面里的世界就跟着你的手走,而不是靠打字描述。
📄 原文摘要(英文)
Interactive video world models enable exploration through text and camera controls, yet direct object interaction requires finer control. Hands provide a natural interface, but translating continuous hand movements into coherent visual responses remains challenging because of complex hand-object dynamics and demanding real-time computation. To address these challenges, we introduce LynnReal-Hand, a hand-controlled video world model that unifies precise interaction modeling with efficient streaming generation. We curate 100K interaction clips spanning diverse real and synthetic environments and develop hand-aware supervision to improve fine-grained hand motions and object interactions. Building on this foundation, we learn a causal flow map for few-step generation and introduce region-aware score distillation that combines complementary video and image priors to refine interaction dynamics and visual appearance. To further reduce inference latency, we propose Ultra-Fast-VAE, a compact video decoder that preserves high reconstruction fidelity with substantially lower computational cost. Extensive evaluations on HandBench demonstrate superior hand-following accuracy and visual quality compared with existing approaches. Our full-sequence model achieves an FVD of 181.67 and a MANO Dice score of 93.02, while the three-step causal model maintains a MANO Dice score of 92.49. For real-time interaction, 540p streaming on a single H200 reaches 18.03 FPS with 741ms latency and an optimized peak GPU memory footprint of only 34.89GB. We will release the datasets, model weights, and the full training and inference code to support research on hand-controlled video world models.