把35B大模型塞进24GB显卡，靠的是提前猜下一层
大模型跑不动，通常怪显存不够。这篇反着来：它把模型拆开，大部分权重放在硬盘上，只把当前要用的部分读进显存。但有个死结——MoE模型每一层要先算完这一层，才知道下一层该读哪些专家，硬盘读取来不及。Edge0的做法是训练一个轻量预测头，提前一层猜出下一层的路由结果，直接拿预测当路由用，读盘和计算就重叠起来了。代价是预测可能不准，于是再用一个LoRA把精度补回来。结果：35B模型在单张24GB显卡上跑到每秒20个token，峰值显存只用3GB，精度接近原版fp16。它不是你明天就能拿来跑自己模型的东西，但这条“用预测换时间”的思路，可能是消费级硬件跑大模型的下一步。
📄 原文摘要(英文)
Mixture-of-experts (MoE) inference on consumer hardware is bounded by weight memory: a 35B-class model is 19.5GB at 4-bit, and sparsity shrinks the compute per token, not the bytes that must be held. Naive offloading to SSD does not help on its own, because layer N+1's experts must be chosen before layer N's output exists, so the reads cannot start early enough to hide behind compute. We present Edge0, a streaming MoE inference engine that closes the gap with a prerouter: a per-layer head predicts the next layer's routing one token ahead, and the prediction is consumed as the routing itself, so the staged expert set equals the routed set and nothing is dropped. An unmerged recovery LoRA, trained on the student path, pays back the quality lost to int4 quantization and routing replacement. On a single 24GB machine, Edge0 serves a 35B MoE at 20tok/s inside 3GiB of peak active memory, within a few points of its fp16 teacher on average across five public benchmarks. An 8B tier runs on the same framework, and the framework, checkpoints, and adapters are open source.