长文本注意力机制提速28倍，MiniMax开源稀疏注意力
大模型处理超长文本（比如整本小说或代码库）时，注意力机制的计算量会随文本长度平方增长，导致速度极慢。MiniMax 提出了一种稀疏注意力方法：不是让模型关注所有词，而是先快速筛选出最重要的几个关键块，只对这些块做精确计算。在 1090 亿参数的模型上，处理 100 万 token 时，每个 token 的注意力计算量减少了 28.4 倍，实际推理速度在 H800 GPU 上提升了 14 倍（预填充阶段）和 7.6 倍（生成阶段），且模型性能没有下降。这不是你明天就能直接用的工具，但它意味着未来长上下文 AI 应用（如智能体、代码分析、持久记忆）可能变得实用。
📄 原文摘要(英文)
Ultra-long-context capability is becoming indispensable for frontier LLMs: agentic workflows, repository-scale code reasoning, and persistent memory all require the model to jointly attend over hundreds of thousands to millions of tokens, yet the quadratic cost of softmax attention makes this untenable at deployment scale. We introduce MiniMax Sparse Attention (MSA), a blockwise sparse attention built upon Grouped Query Attention (GQA). A lightweight Index Branch scores key-value blocks and independently selects a Top-k subset for each GQA group, enabling group-specific sparse retrieval while maintaining efficient block-level execution; the Main Branch then performs exact block-sparse attention over only the selected blocks. Designed around a principle of simplicity and scalability, MSA is deliberately streamlined, making it straightforward to deploy efficiently across a broad range of GPUs. To translate sparsity into practical speedups, we co-design MSA with a GPU execution path that uses exp-free Top-k selection and KV-outer sparse attention to improve tensor-core utilization under block-granular access. On a 109B-parameter model with native multimodal training, MSA performs on par with GQA while reducing per-token attention compute by 28.4x at 1M context. Paired with our co-designed kernel, MSA achieves 14.2x prefill and 7.6x decoding wall-clock speedups on H800. Our inference kernel is available at: https://github.com/MiniMax-AI/MSA. A production-grade natively multimodal model powered by MSA has been publicly released at: https://huggingface.co/MiniMaxAI/MiniMax-M3.