KV缓存淘汰：随机抽签竟和最聪明策略打平
大模型想长链条推理，得把中间思考过程全记在缓存里，这缓存大得离谱，成了内存瓶颈。之前所有压缩办法都是一个思路：给每个缓存token打分，预测它以后重不重要，只留高分。这篇论文说，那个打分信号几乎没用——他们干脆完全不打分，在每个注意力头里均匀随机淘汰，结果在4个模型、6个推理任务上，和之前最强的淘汰策略打成平手，部署时吞吐量还高32-43%。为什么随机能赢？因为推理过程自带两层冗余：模型在思考时会不断复述自己还需要的东西，而且每个注意力头都各存一份副本。所以只要把最脆弱的提示词部分保住，剩下的随机抽，总能抽到够用的副本。这不是你明天能用的技巧，但它改写了我们对KV缓存压缩的认知：之前所有聪明策略的复杂度，可能都花在了根本不重要的地方。
📄 原文摘要(英文)
Large language models achieve superior performance on tasks that require extended reasoning, but long chains of thought make the KV cache a severe memory bottleneck. Existing KV cache compression methods share one paradigm: score each cached token by some estimate of how much it will matter later, and keep the top-scoring ones. We show that the selection signal contributes almost nothing. Random Attention keeps the prompt and evicts uniformly at random within each attention head, computing no score at all; across four models and six reasoning tasks it matches the strongest prior evictor while serving 32-43% higher throughput than it in vLLM deployment. Controlled experiments explain this by showing that 1) the prompt is the fragile part of the cache, and most of the gap between selectors is just whether their selection signal happened to keep it; 2) the reasoning trace protects itself against eviction with redundancy at two levels, in the text (the model restates what it still needs as it works) and across attention heads (each keeps its own copy of the trace), so once the prompt is safe, a random draw retains enough copies of what the model still needs, and no score is required to pick them. Our code is publicly available at https://github.com/SalesforceAIResearch/Random-Attention.