大模型内存砍到1/1000，性能只掉一成
大模型里有一块常被忽略的内存大户：每层嵌入向量，在最新的 Gemma 4 里占了 35% 的参数。传统压缩到这种程度，每个 token 只剩几个字节，基本等于把信息扔了。这篇换了个思路：不再给每个位置单独存压缩码，而是让所有层共享同一套码，每层再用自己的小解码器还原。更关键的是，它不追求还原原向量，而是直接拟合模型的输出路径——只要最终答案对就行。结果：压到 1/1000，七个基准平均只掉 16% 性能；压到 1/100，掉 2%，推理时 GPU 内存省三成，速度几乎不变。这不是你明天能用的工具，但它指向一个趋势：模型越来越大的同时，内存瓶颈可能比算力更早到来，而压缩的思路正在从「尽量还原」转向「只保结果」。
📄 原文摘要(英文)
Scaling embedding capacity alongside Transformer computation is an emerging trend in language models. This trend makes high compression rates increasingly useful. Hence, we study extreme compression of Per-Layer Embeddings (PLE), which follow this trend and account for 35.2% of the parameters of Gemma 4 E4B. At such rates, conventional quantizers give each token-layer vector its own code of only a few bytes. We instead develop OPTIC (Output-Path-fitted, Token-shared Indexed Codes), which stores one quantized code tuple per token and decodes it at every layer with a layer-specific affine decoder. Codes and decoders are fitted to the PLE output path rather than to the stored vectors. At about 100times PLE compression of bfloat16 (BF16) E4B, OPTIC retains an aggregate 97.8% of the uncompressed model's performance across seven benchmarks, whereas product quantization, the strongest of six conventional baselines, retains 66.3%. At 594times and 1,002times PLE compression of BF16 E4B, OPTIC retains 90.5% and 84.1%. On the official E4B release with 4-bit linear weights and 16-bit activations (W4A16), OPTIC retains 96.9% at about 100times PLE compression. During inference at about 100times PLE compression, BF16 E4B uses 29.7% less GPU memory than with uncompressed PLE at the same key-value cache size, while running at most 0.40% slower.