视频世界模型：把3D记忆藏在潜空间
视频世界模型要生成连贯的3D场景，传统做法是把画面转成点云存起来，每次生成新视角都要把点云渲染成像素图、再编码回潜空间，既慢又丢信息。这篇论文直接把3D场景信息存在扩散模型的潜空间里——用深度信息把潜空间里的特征点投射到3D位置，查询时直接在潜空间里做视角变换，绕开了像素空间。结果：生成速度提升10倍以上，内存占用减少55倍，而且画面质量还更好。它不是你明天能用上的，但如果你关心AI视频生成或3D世界模型，这是当前最优雅的架构思路。
📄 原文摘要(英文)
Video world models that maintain 3D spatial consistency across generated frames typically rely on explicit point cloud memory constructed in RGB space. This design is both computationally expensive, requiring repeated rendering and VAE encoding, and inherently lossy, as the round trip through pixel space discards rich features of the learned latent representation. In this paper, we introduce latent spatial memory for video world models, a persistent 3D cache that stores scene information directly in the diffusion latent space, avoiding pixel-space reconstruction. Building on this, we propose Mirage, a latent-space spatial memory framework that constructs the memory by lifting latent tokens into 3D via depth-guided back-projection and queries it by synthesizing novel views through direct latent-space warping. This unified formulation eliminates both the information loss of pixel-space reconstruction and the computational burden of repeated encoding and rendering. Experiments show that latent spatial memory achieves up to 10.57times faster end-to-end video generation and 55times reduction in memory footprint relative to explicit 3D baselines. Leveraging the geometric prior of the diffusion model, Mirage attains state-of-the-art performance on WorldScore and strong reconstruction quality on RealEstate10K.