线性注意力被一个老方法反超了
大模型每多读一个词，记忆成本就涨一点，线性注意力被当成解药吹了很久。这篇论文拿它跟一个更简单的老办法比：滑动窗口注意力，只在附近一小段里找信息，再在开头放几个“锚点”兜底。结果在长文本推理任务上，老办法的准确率是线性注意力的2到10倍，而且不用重新训练、内存更低。线性注意力不是不行，是它得从头训练或大量微调才配得上这个名声。这不是你明天能用上的技术，但它提醒你：AI圈子里“新方法”的标签，常常没跟“老方法”认真比过。
📄 原文摘要(英文)
Due to the nature of quadratic attention, Large Language Models (LLMs) consume a lot of memory and energy. Every new token costs more than the previous one. For each additional token, the keys and values must be stored in memory indefinitely, which is unsustainable. Several alternatives have been proposed to fix the quadratic scaling problem, one of which is retrofitting LLMs to use Linear Attention. This idea has attracted a lot of attention, given its promise to solve the quadratic scaling problem with state-of-the-art performance at low cost. However, this line of research has not been properly compared to simpler baselines. In this work, we show that Sliding Window Attention (SWA) with sinks performs as well or better than post-trained Linear Attention models. We observe this across multiple LLMs on various downstream tasks. For long-context reasoning tasks (Needle-in-a-Haystack and BABILong), SWA achieves massively higher performance (2 to 10 times higher than linear attention). SWA requires no post-training, is extremely fast, and requires low memory; therefore, making it an extremely cheap and reliable solution. To reduce inference memory cost, we strongly recommend switching to SWA instead of post-training linear models. Linear attention models may have shown some promise, but they likely require to be trained from scratch or extensive post-training in order to even match SWA.