AI看视频学3D：盯住一个点，就能生成新视角
你刷到一个视频，想换个角度看——比如从侧面看一个人跳舞。现在的AI要么重建3D模型但动态物体容易变形，要么直接生成画面但动作会飘。这篇论文发现，AI模型里某些注意力层其实已经知道“这个点在下一帧应该在哪”，只是没用好。于是他们加了一个“点追踪”任务：让模型在生成新视角时，同时预测每个像素点在后续帧和不同视角下的位置，并把这个预测结果作为监督信号。训练后，模型生成的新视角视频里，物体的形状和运动都更稳了。它不是你明天能用上的，但方向很明确：让AI理解视频里的“点”怎么动，是生成逼真新视角的关键。
📄 原文摘要(英文)
Synthesizing a novel-view video from a monocular reference video along a target camera trajectory requires both geometric consistency and motion fidelity with respect to the reference video. Existing methods based on explicit 3D representations are limited by the accuracy of off-the-shelf reconstruction modules, which often produce inaccurate geometry for dynamic objects in monocular videos. In contrast, camera-conditioning-only methods can achieve high visual quality but often struggle to preserve geometric and motion consistency. In this work, we introduce MVTrack4Gen (Multi-View point Tracking for Novel-View Generation), a motion-aware training framework that leverages multi-view point tracking as an additional geometric and motion supervision signal for camera-conditioning-only novel-view video diffusion models. Our key finding is that specific attention layers encode strong correspondence cues, where query features attend to key features at geometrically corresponding locations across views and over time, and the misalignment of these correspondences causes motion inconsistency. Based on this observation, we route these features into an auxiliary multi-view tracking head and jointly train the diffusion model with a point-tracking objective. By explicitly strengthening these motion-aware correspondences, MVTrack4Gen improves existing models to better follow the motion in the reference view and maintain cross-view geometric consistency. Across diverse benchmarks, our method achieves state-of-the-art geometric consistency and competitive camera accuracy.