视频生成的新思路：先懂语义，再谈压缩
视频生成模型一直靠自编码器把视频压成紧凑的“潜空间”，但传统做法只盯着像素还原，压缩出来的空间缺乏高层语义组织。V-RAE 换了个思路：先用现成的视觉基础模型提取语义特征，再在这个语义空间上做轻量压缩和重建。结果在 K600 上重建质量超过所有大预训练视频自编码器，语义信息保留更多，生成质量更好，收敛还快 6 倍。它不是你明天能直接用的工具，但指向一个趋势：生成模型的底层表示，正在从“像素忠实”转向“语义优先”。
📄 原文摘要(英文)
Latent video generation relies on autoencoders to define a compact space in which generative models operate. Although video autoencoder architectures have evolved substantially, their latent spaces are still optimized primarily for pixel-level reconstruction and provide limited high-level semantic organization. A reconstruction-optimal latent space, however, need not be well suited to generative modeling. We propose V-RAE, a video representation autoencoder that builds compact generative latents on top of frozen vision foundation model representations. A lightweight temporal pooling module removes temporal redundancy while preserving semantic structure, and a video decoder reconstructs continuous motion from the compressed features. We evaluate V-RAE with four representative frozen encoders on video reconstruction, semantic probing, and class-conditional generation. V-RAE achieves 2.13 rFVD on K600, outperforming all evaluated large-scale pretrained video VAEs. Its latents retain substantially more semantic information than conventional video tokenizer latents. Under matched generation settings, our best variant achieves gFVD scores of 117.86 and 19.16 on UCF101 and K600, respectively, while converging up to 6x faster}. We further show that reconstruction quality alone is insufficient to characterize generative utility and introduce tFVD, a temporal-coherence diagnostic that correlates more reliably with downstream generation quality. Beyond video generation, V-RAE also improves future video prediction on Cityscapes over the Wan 2.2 VAE latent space under matched prediction settings. Taken together, the experiments show that frozen semantic representations can support video reconstruction, generation, and predictive modeling. The project page: https://v-rae.github.io/.