把35B大模型塞进24GB显卡，靠的是提前猜下一步
大模型跑不动，瓶颈往往不是算力，而是显存装不下参数。这篇把 35B 参数的混合专家模型(MoE)塞进 24GB 显卡，峰值只占 3GB，速度还有 20 token/秒。做法很反直觉：它不老老实实等模型算完一层再读下一层，而是训练了一个「预路由」小头，提前一步预测模型下一步会调用哪些专家，直接拿预测结果当真实路由用，这样就能提前从 SSD 读数据，把慢速硬盘藏在计算后面。代价是预测偶尔不准，但作者用一个小 LoRA 适配器把精度损失补了回来，五个公开基准上只比原版 fp16 模型差几个点。它不是你明天就能直接用的产品，但「用预测掩盖延迟」这个思路，可能是消费级硬件跑大模型的下一步。
📄 原文摘要(英文)
Mixture-of-experts (MoE) inference on consumer hardware is bounded by weight memory: a 35B-class model is 19.5GB at 4-bit, and sparsity shrinks the compute per token, not the bytes that must be held. Naive offloading to SSD does not help on its own, because layer N+1's experts must be chosen before layer N's output exists, so the reads cannot start early enough to hide behind compute. We present Edge0, a streaming MoE inference engine that closes the gap with a prerouter: a per-layer head predicts the next layer's routing one token ahead, and the prediction is consumed as the routing itself, so the staged expert set equals the routed set and nothing is dropped. An unmerged recovery LoRA, trained on the student path, pays back the quality lost to int4 quantization and routing replacement. On a single 24GB machine, Edge0 serves a 35B MoE at 20tok/s inside 3GiB of peak active memory, within a few points of its fp16 teacher on average across five public benchmarks. An 8B tier runs on the same framework, and the framework, checkpoints, and adapters are open source.