MoE模型路由器的数学升级：让专家更精准
混合专家模型（MoE）靠一个“路由器”决定把输入分给哪些专家处理。但现有路由器设计没有数学原则来保证它真正理解专家。这篇论文提出一个原则：让路由器的每一行对齐对应专家矩阵的“主奇异方向”——这是矩阵最核心的数学特征。他们用“流形幂迭代”方法实现对齐，在1B到11B参数的模型上验证了效果提升。这不是你明天能直接用的技术，但它为MoE模型提供了一种更严谨的设计思路，可能影响未来大模型的架构。
📄 原文摘要(英文)
Router is the cornerstone component to the Mixture-of-Experts models. Serving as expert proxies, the rows of the router matrix compute their similarity to the MoE inputs to determine which subset of experts is activated. Ideally, each router row is designed to encode the expert matrix into this representative vector, such that its dot-product with token can better reflect token-expert affinity. However, there exists no design principles to enforce this condensation. In this paper, we propose to align each router row with the principal singular direction of the associated expert, as this direction provides the most expressive mathematical description of a matrix. Based on this principle, we propose a router redesign with Manifold Power Iteration (MPI). Specifically, it introduces a "Power-then-Retract" paradigm, where a power iteration step is performed on the router weights, followed by a retraction to impose a norm constraint to ensure both efficiency and stability. Theoretically, we show that MPI drives router rows to converge toward the principal singular directions of associated experts. Empirically, we pretrain MoE model across scales from 1B to 11B parameters to confirm that this alignment facilitates more effective MoE models.