AI模型新架构：把信息通道从4条扩到16条，效果更好还更省算力
Transformer模型有个隐藏的瓶颈：残差流（信息在主干的通路）太窄。之前有人提出Hyper-Connections，把残差流从1条扩成4条并行通道，效果提升明显。但再往上扩（比如16条）就失效了——性能不涨，训练成本却暴涨。这篇论文找到了两个原因：写回信息不足（通道多了，但每个通道写回主干的信号太弱）和残差混合计算成本随通道数立方增长。他们提出xHC，用两个技巧解决：一是给写回信号做时间增强（让每个通道写回的信息更丰富），二是只更新16条中的4条，但保留对所有16条状态的完整访问。在18B和28B参数的MoE模型上，xHC比之前的mHC平均提升4个点，而且达到同样损失所需的算力更少（mHC需要1.19倍，原始模型需要1.5倍）。他们还做了xHC-Flash优化内存，让16通道的内存开销和之前4通道差不多。这不是你明天能直接用的技术，但它揭示了一个趋势：模型性能的下一波提升可能来自“拓宽信息通路”而非单纯堆参数。
📄 原文摘要(英文)
Hyper-Connections (HC) expand the residual stream of Transformers into N parallel streams, providing a form of memory scaling beyond model width and depth. Manifold-Constrained HC (mHC) stabilizes this formulation at scale. The large gains from N{=}1 to N{=}4 suggest residual-stream expansion as a promising scaling axis. However, existing HC-family methods typically stop at N{=}4. Our experiments reveal why: scaling mHC beyond this point yields diminishing performance gains and rapidly increasing training cost. We attribute this limitation to two bottlenecks: insufficient write-back information for an expanding number of streams and residual-mixing generation whose cost scales cubically with N. To address both bottlenecks, we propose xHC (Expanded Hyper-Connections), the first HC-family method to achieve meaningful expansion beyond N{=}4. xHC combines temporal feature augmentation for richer write-back with a sparse residual-stream architecture that updates only k=4 of the N=16 streams while retaining dense access to the full residual state. Across 18B and 28B MoE models, xHC delivers strong and consistent downstream improvements. On an 18B MoE model, xHC improves the average downstream score by 4.0 points over mHC, while adding only modest training FLOPs over the vanilla baseline. Scaling-law experiments show that the vanilla and mHC require 1.50times and 1.19times the compute of xHC, respectively, to reach the same loss. Practical large-N training also requires controlling memory traffic from the expanded residual state. We therefore introduce xHC-Flash, which reduces the per-sublayer memory traffic from 73.5C to 40C, comparable to the 34C required by mHC at N{=}4, while retaining the gains of full xHC. Together, xHC and xHC-Flash make large-N residual-stream expansion effective and practical for LLM pre-training.