AI推理时，内存不够？新方法只留关键记忆
大模型在生成回答时，需要记住之前的所有内容（KV缓存），上下文越长，内存消耗越大，甚至导致卡顿。现有方法靠固定规则或粗略打分来丢弃旧记忆，但容易误删重要信息。KVpop让模型自己学习哪些记忆该留：它用“未来注意力”作为训练目标——即预测未来哪些词会回头看当前记忆，从而精准保留关键部分。在数学推理测试中，Qwen3-4B模型只保留25%的记忆，性能仍达到完整记忆的98%；压缩到12%时仍有97%。这不是你明天能直接用的工具，但它指向一个趋势：AI推理不再需要“记住一切”，而是学会聪明地遗忘。
📄 原文摘要(英文)
Key-value (KV) cache growth is a major bottleneck in autoregressive decoding, as memory and bandwidth scale linearly with context length. Existing KV eviction methods often rely on static heuristics or proxy scores, which poorly track future token utility and cause brittle eviction as relevance shifts. To address this, we introduce KVpop, which learns a fixed-budget KV eviction policy by directly supervising the keep-or-drop decision. The scorer is trained against a novel future-attention target, computed efficiently without materializing dense attention maps. We further introduce a delayed memory-based scorer that, uniquely among learned eviction methods, defers scoring for a fixed number of steps to exploit near-future context. On AIME and HMMT mathematical reasoning, KVpop retains 98% of full-attention performance on Qwen3-4B at 75% KV cache compression and 97% at 88% compression, consistently outperforming established eviction baselines. Qwen3-8B shows even stronger results, reaching near-full teacher performance. These results show that supervising eviction with future-attention signals cuts memory costs while maintaining quality.