AI推理提速：让MoE模型专家就近分配
大模型推理时，每个请求会激活不同的“专家”模块。现有路由只考虑负载均衡，但负载相同不等于速度相同——因为不同专家权重加载耗时差异大。ELDR通过预填充阶段的专家激活模式预测后续请求会用到哪些专家，然后离线用K-means将专家签名空间分区，在线将请求路由到最匹配且负载最低的worker。在vLLM上测试，40GPU规模下中位TPOT降低5.9-13.9%，且模型输出不变。这不是你明天能直接用的工具，但它揭示了MoE推理优化的新方向：从负载均衡转向专家局部性感知。
📄 原文摘要(英文)
In prefill-decode (PD) disaggregated LLM serving, each request is assigned to a decode worker after prefill. Existing decode routers balance only load; for mixture-of-experts (MoE) models this is incomplete: equally loaded workers can differ in latency, since each decode step loads the weights of every distinct expert its batch activates. We present ELDR, an expert-locality-aware decode router for PD-disaggregated MoE serving. From a request's prefill expert activations, ELDR builds an expert signature predicting the experts it will activate during generation. Offline, balanced K-means partitions signature space across decode workers; online, locality-band routing sends each request to the least-loaded worker among those best matching its signature. A signature cache, co-indexed with the KV cache at KV-block granularity, keeps signatures exact under prefix caching. Implemented in vLLM and evaluated on deployments of up to 40 GPUs, ELDR reduces median TPOT by 5.9-13.9% over the strongest of four load-balancing baselines across three MoE models and two workloads, with model outputs unchanged.