视频生成服务不再卡顿：TurboServe 让多用户流畅看片
你刷短视频时，AI 正在一帧帧生成画面，但服务器可能同时服务几百人，导致你等很久。TurboServe 是首个专为这种“流式视频生成”设计的调度系统，它像智能交通指挥：把长时间运行的视频任务在 GPU 间动态迁移，避免某块 GPU 堵死；同时根据用户数量自动增减 GPU 数量，闲时省钱、忙时加力。在真实生产数据上，最差情况下的等待时间降低 37.5%，GPU 成本也降了 37.2%。虽然你明天用不上，但它是视频生成服务走向大规模商用的关键一步。
📄 原文摘要(英文)
Streaming video generation is emerging as a new serving workload in which users interact with long-lived sessions that generate video progressively, chunk by chunk. Unlike offline video generation or typical LLM serving, streaming video generation must preserve session state across active and idle periods, repeatedly schedule ongoing sessions, and deliver each chunk under a tight latency target. This creates two key serving challenges in multi-user, multi-GPU environments: session duration heterogeneity, where long-running sessions make placement decisions suboptimal over time, and temporal user-demand heterogeneity, where the number of active sessions fluctuates sharply across bursts and idle periods. We present TurboServe, the first serving system designed specifically for streaming video generation workloads. TurboServe formulates serving as an online scheduling problem that jointly coordinates session placement and GPU provisioning. Its closed-loop scheduling algorithm combines a migration-aware placement controller, which rebalances sessions across GPUs to reduce the maximum per-chunk latency, with a load-driven autoscaling controller, which adapts the GPU budget to workload variation for improved cost efficiency. To support these decisions at runtime, TurboServe implements coalesced chunk processing for batching concurrent active sessions on the same GPU, GPU-CPU offloading for session suspension and resumption, and NCCL-based GPU-GPU migration for online rebalancing. We evaluate TurboServe on real-world production traces from Shengshu Technology across multiple model sizes and GPU clusters with up to 64 NVIDIA B300 GPUs. Compared with baseline serving configurations, TurboServe reduces worst-case per-chunk latency by 37.5% and total GPU operating cost by 37.2% on average. Our code is publicly available at https://github.com/shengshu-ai/TurboServe.