让大模型读超长文本，效率反超全量注意力
大模型处理长文本时，传统注意力机制的计算量随文本长度平方增长，就像每读一个字都要回头翻一遍全书。现有稀疏注意力方法虽然省力，但选错重点段落，效果打折。这篇提出分层稀疏注意力：模型先快速定位关键段落，再对每个段落独立做精细阅读，最后按重要性融合信息。关键是，这个定位能力是直接通过语言模型训练学出来的，而不是靠外部规则。结果：在训练长度内效果不输全量注意力，还能外推到训练长度64倍以上的文本，检索准确率90%。更实用的是，现有模型只需轻量继续训练就能转换，保留原有能力的同时获得超长文本处理能力。它不是你明天就能直接用的工具，但指明了长上下文模型的方向：效率与效果可以兼得。
📄 原文摘要(英文)
Scaling modern large language models (LLMs) to long contexts is limited by the quadratic computation cost, and poor length extrapolation of dense attention. Chunk-wise sparse attention offers a promising alternative, but all existing methods fall short of full attention because of their inaccurate chunk selection. We propose Hierarchical Landmark Sparse (HiLS) Attention, a chunk-wise sparse attention mechanism that learns chunk selection end-to-end under the language-modeling (LM) loss. HiLS factorizes attention hierarchically: each query performs attention independently with each retrieved chunk to extract chunk-specific information, and the resulting outputs are fused according to chunk retrieval scores. By incorporating retrieval scores into the forward attention computation, HiLS optimizes them directly with the LM loss, enabling end-to-end retrieval learning and native sparse training. Experimental results show that HiLS-Attention achieves performance comparable to, and in some cases better than, full attention at in-domain context lengths. Meanwhile, HiLS-Attention extrapolates more than 64times the training context length with 90% retrieval accuracy, far beyond full attention. Moreover, existing full-attention models can be converted to HiLS-Attention with lightweight continued pretraining, preserving in-domain performance while acquiring ultra-long-context extrapolation. Together with its sparse KV access and computation, HiLS-Attention breaks the usual efficiency-performance trade-off, enabling long-context LLMs that are both more efficient and more effective on general long-context tasks than their full-attention counterparts.