注意力头也搞专家分工，省一半算力不掉分
Transformer 的注意力机制有个浪费：不管简单还是复杂 token，都用同样多的注意力头去处理。这篇把「专家混合」的思路塞进注意力层——每个 token 只激活一半的查询头，由路由器按 token 内容动态选，而键值头保持全量。250M 参数模型在 300 亿 token 上训练，下游准确率持平，但查询头计算量减半。它不是你明天能用上的，但指向一个趋势：大模型正在从「全员上阵」转向「按需调度」，省下的算力可以塞更多参数或更长上下文。
📄 原文摘要(英文)
Self-attention is central to Transformer performance and is often the most expensive part of the Transformer at long context lengths because its pairwise token interactions scale quadratically with sequence length. Standard dense attention also applies the same set of attention heads to every token regardless of token difficulty or information content. This uniform activation can waste compute, especially as sequences grow longer and attention cost increases rapidly. We propose Grouped Query Experts (GQE), a mixture-of-experts layer on top of grouped-query attention (GQA). Within each GQA group, a router selects k query-head experts per token while all key-value (KV) heads remain dense and unchanged. Thus, GQE keeps the KV cache benefits of GQA and reduces only the active query-head computation. On a fixed 30B token budget at the 250M parameter scale, GQE matches the all-active GQA baseline in downstream accuracy while activating half the query heads per token.