给大模型外挂一个可扩容的记忆硬盘
大模型把记忆和推理揉在同一套参数里，想加记忆就得整个模型重训，成本极高。这篇把记忆单独拆出来，做成一个可独立扩容的模块，像给电脑加硬盘一样：主模型只有4亿参数，外挂一个69亿参数的记忆模块，综合成绩反而超过了120亿参数的完整大模型，参数还少了39%。在17项基准测试上，这个组合平均分从29.86涨到37.34，而单独把主模型从4亿扩到120亿，只能到37.24。换句话说，与其把整个模型做大，不如单独把记忆做大，性价比更高。这不是你明天能直接用的技术，但它指向一个趋势：未来的AI可能不再靠一味堆参数，而是靠模块化组合，像攒机一样按需配置。
📄 原文摘要(英文)
Decoder-only language models entangle long-term memory and reasoning in a single parameter set, making it difficult to scale memory capacity independently. Memory Decoder introduces a parametric long-term memory module but only studies it at a relatively small scale. In this work, we present Memory Decoder at Scale, scaling memory models up to 6.9B parameters and pretraining them on 300B tokens. At this data scale, the combined cost of indexing and search makes a standard Faiss pipeline infeasible. We address this bottleneck with a distributed pipeline for Faiss indexing and retrieval, together with sparse, batch-wise loading of kNN distributions. Across model scales, we find that allocating more parameters to memory yields a better parameter-performance tradeoff than scaling the base model alone. On 17 benchmarks, pairing a 6.9B general memory with Pythia-410M raises its average score from 29.86 to 37.34, surpassing Pythia-12B (37.24) with 39% fewer total parameters. For Qwen3 Base models ranging from 0.6B to 14B, 1.7B domain memories improve the average score across the three domains by more than 9 points at every scale. Overall, our results demonstrate that independently scaling pretrained memory offers a more parameter efficient path to improving language model performance.