AI Pulse
📄 论文解读

让AI手写神经网络,它弃了Transformer

当大模型不再靠梯度下降训练、而是直接手写神经网络的权重时,它自己选的结构不是Transformer,而是一张稀疏的查表。研究者让三个前沿大模型在只有10个标注样本的沙盒里,要么生成固定架构的权重、要么连架构一起设计。结果三个模型都自发把事实编译成一张表,用固定程序去查,最高到54.4%的测试准确率——比同一个大模型手写的最准Transformer还高,参数却少了约1800倍。更妙的是这张表可解释:每个权重有含义,答错时能定位到具体某一行,删掉一行只影响一个答案。这不是说大模型比预训练模型更准——预训练模型仍然更强——而是说,当训练方式变了,结构也该换。它不是你明天能用上的东西,但它动摇了「Transformer是唯一答案」的默认前提。

📄 原文摘要(英文)

Standard neural architectures such as the Transformer were chosen because they train well with gradient descent and run fast on GPUs. In weight-space programming, a large language model (LLM) writes target weights by hand without gradient-based training, so those reasons no longer apply. Should we keep the Transformer? Given minimal guidance, ten labeled examples, and a sandbox, mainly on multiple-choice questions, we task three frontier LLMs with either generating the weights of a fixed architecture or designing the architecture themselves. When free to design, all three compile plain-text facts into a sparse table queried by a fixed procedure, reaching up to 54.4% test accuracy. An LLM-designed model achieves higher test accuracy than the most accurate released Transformer that the same LLM wrote while using about 1,800x fewer parameters. Without gradients, they choose a table, in which the LLM can state what each weight means and can find and rewrite a wrong one. Each wrong answer points to one row, and removing a row changes about one other answer. From this, we propose an associative-memory-based model up to 17x faster on GPUs than a same-sized Transformer. We also test pretrained weights, other benchmarks, and tasks where models generate answers. Pretrained LLMs remain more accurate; the claim concerns which structure to build when writing weights.

arXiv 原文

订阅 AI Pulse

每天 08:00 · 12:30 · 18:30 · 23:50 更新