AI代理跑得慢，问题不在模型
同一个模型、同一批 GPU，换一套调度方式，请求延迟最多降 4.1 倍，首字等待时间(P95)降 14 倍。卡没变，模型没变，变的是「谁先算、谁后算、缓存留什么」。
现在服务大模型的通行做法，是把「读题」(prefill)和「写答案」(decode)拆到不同机器上跑。但 AI 代理类任务——写代码、解数学、操作电脑——每一步长短不一，还会反复回头用同一段上下文。固定拆法会让一边闲着一边堵着，还会把代理马上要复用的缓存清掉。
这篇让机器在运行时自己换角色：算力分配和缓存管理拆开管，节点能在读题和写答案之间灵活切换；一个在线规划器按当前任务构成决定怎么分；缓存该清谁、该搬去哪，则听代理上下文管理的信号。在 GLM-5.2 和 Kimi K2.6 上、最多 256 块 GB300 上测了编码、数学、电脑操作三类任务，对比的是 NVIDIA 的生产级框架 Dynamo。
它不是你明天能装来用的东西——这是给跑推理集群的人看的。但如果你在用 AI 代理干活、觉得它「想一下卡一下」，这篇说明那口气有一大截憋在调度上，不在模型里。
📄 原文摘要(英文)
Agentic workloads are a major source of LLM inference demand. Existing prefill--decode (PD) disaggregated serving systems offer limited flexibility for runtime adaptation of compute allocation and KV cache management, and rely on coarse signals that only partly capture the diverse and dynamic agent workload. These limitations lead to load imbalance between prefill and decode nodes and eviction of KV caches that agents will reuse. We present Evolving Infra, an agent-adaptive serving framework that adapts compute allocation to agent workloads and KV cache management to agent context management at runtime. We decouple compute allocation from key--value (KV) cache management, allowing nodes to switch efficiently between prefill and decode. Building on this design, an online planner matches the PD split to workload composition, while an agent-aware distributed KV manager guides cache eviction and transfer with signals from agent context management. Across coding, math, and computer-use tasks with GLM-5.2 and Kimi K2.6 on up to 256 GB300 GPUs, Evolving Infra reduces mean end-to-end request latency and P95 TTFT by up to 4.1x and 14x over Dynamo, NVIDIA's production framework.