用哈希签名替代词表，大模型参数省了但更强了
大模型通常需要一张巨大的词表来记住每个词，词越大模型越胖。这篇论文反其道而行：把每个词变成一串短的哈希签名（类似指纹），用多个哈希函数生成，再通过专门的编码器和解码器处理。结果在1亿到30亿参数规模上，不仅参数更少，性能反而超过传统Transformer。更意外的是，它支持多语言扩展时参数不增加——加新语言不用加词表。这不是你明天能用的技术，但它指向一个趋势：未来大模型可能不再靠死记硬背词表，而是靠巧妙的压缩表示来兼顾效率和能力。
📄 原文摘要(英文)
Language models (LMs) represent tokens using embedding matrices that scale linearly with the vocabulary size. To constrain the parameter footprint, prior work proposes hashing many tokens into a single vector within encoder-only models. While this offers parameter efficiency, many-to-one collisions prevent its use in causal LMs. In this paper, we propose MultiHashFormer, a new framework that allows hash-based autoregression. Each token is represented as a unique hash signature, a short sequence of discrete hash IDs, generated by multiple independent hash functions. A Hash Encoder compresses this signature into a single latent vector for processing by a Transformer decoder. Then, a Hash Decoder generates the hash signature of the next token, which is then mapped back to text. We evaluate our approach at the 100M, 1B and 3B parameter scales, demonstrating that MultiHashFormer consistently outperforms standard Transformer LMs across multiple benchmarks. Furthermore, we show that our model handles multilingual vocabulary expansion with a constant parameter footprint without any modifications.