视频生成快到能实时互动了
视频生成模型慢到没法边动边看,这篇把瓶颈找了出来:大家以为只是实现细节的 VAE 解码和 KV 缓存更新,竟占了端到端延迟的一半以上。他们干脆把这些「必需品」换成轻量替代,1.3B 版本在单张 B200 上做到每帧 36 毫秒、每秒 28 次响应,画质追平一个慢近 8 倍的现有最强模型;14B 版本用 8 卡节点每帧 45 毫秒,真实感还更好。演示里,镜头随你动、物理随你改的视频都能实时生成。它不是你明天能装进手机的东西,但它指向的是:视频生成从「等几分钟出一段」变成「你动它就跟着变」的那种交互。
📄 原文摘要(英文)
Autoregressive video diffusion models generate frames based on new observations and inputs, making them a promising medium for interactive visual experiences. However, existing models are orders of magnitude too slow for fluid interaction. This paper introduces FastFrames, a family of fast autoregressive video diffusion models that achieves interactive latency without compromising quality. We profile the inference pipeline of state-of-the-art models and find that operations often treated as fixed implementation details, such as VAE decoding and KV cache updates, in fact account for over half of end-to-end inference latency. We challenge the assumption that these components are indispensable and replace them with lightweight alternatives. Our 1.3B variant runs at 36 ms per frame, or 28 responses per second (RPS), on a single B200 GPU while matching the generation quality of a state-of-the-art model that is nearly 8x slower. Our 14B variant runs at 45 ms per frame, or 22 RPS, on an 8x B200 GPU node and yields significantly better visual quality and realism. We show that FastFrames enables real-time interaction in applications such as camera-conditioned and physics-conditioned video generation.