MoE新架构：让每个专家都参与，还不增加算力
混合专家模型(MoE)一直有个三角难题：参与、算力、内存三者互相牵制——稀疏路由省算力但只有少数专家干活，密集输出让所有专家参与但算力暴涨。这篇提出 IntBMoE，用一个小型码本把专家拆成「块」，每层用轻量网络把该层所有专家基座合并成一个「合成专家」，路由只选几个块执行，但每个合成专家都吸收了整个池子的知识。参与全满、算力稀疏、内存有界，三个指标第一次能独立调。在图像分类、语言建模、序列推荐上都超过现有 MoE 基线，而且已经部署在高德地图的生成式推荐系统里，服务数亿用户，60ms 延迟内，线上 A/B 测试 UVCTR 相对提升 2.4%。这不是你明天能用的论文，但它是 MoE 从「稀疏凑合」走向「全参与不贵」的一个真实拐点，而且不是实验室玩具——已经在亿级产品里跑着了。
📄 原文摘要(英文)
Mixture-of-Experts (MoE) scales capacity, but existing designs cannot set three quantities independently. For a single token, participation is how many experts contribute knowledge to its output, execution is how many are actually computed (compute cost), and materialization is how many expert-sized parameter sets must be built and stored (memory cost). Sparse routing keeps execution and materialization low, but shrinks participation: for each token, only a few experts contribute. Dense output-mixing restores full participation, but its execution grows with the number of experts. Parameter-merging keeps execution at one expert, but its materialization grows with the number of routing decisions. We propose IntBMoE, a block-conditioned MoE that decouples all three by pairing dense expert composition with sparse block execution. Its blocks come from a small learned codebook, one per entry. At each internal layer, a lightweight hypernetwork merges all expert bases in that layer's pool into one composed expert. Participation is full, because every composed expert draws on the entire pool. Execution stays sparse, because a router sends each token to only a few blocks. Materialization is bounded, because the codebook, not the input, fixes how many blocks exist. Dual-Path Residual Gating (DPRG) further couples two independently composed paths through multiplicative gating. Experiments on image classification show consistent gains over representative sparse and dense MoE baselines. Additional experiments on language modeling and sequential recommendation validate its generalization beyond vision. IntBMoE is fully deployed in AMap's generative recommendation system, serving hundreds of millions of users under a 60ms latency budget, with a 2.4% relative UVCTR gain in online A/B testing. Our code is available at https://github.com/AMAP-ML/DreamX-Rec/.