大模型其实没在“想”,只是懒得想
预训练大模型处理长链条推理时,只用了自己深度的一小截——13个基础模型平均只能可靠追踪1.4到3.6行引用,再深就断。但研究者发现,在某一层插入一个极小的LoRA(仅8秩,所有原参数冻结),Qwen3-8B的24行链式推理准确率从15.5%飙到99%,训练更久还能追到50行;Ouro-1.4B甚至能到160行。机制像接力:LoRA让程序行把“链身份”传给中间层,冻结的注意力头再逐层往上游读,删掉父行注意力就断。它不是你明天能用上的,但它在说:大模型不是能力不够,是默认懒得想——一个微小的外部开关就能唤醒它本有的深度。
📄 原文摘要(英文)
Pretrained transformers use little of their depth to follow references in context. Thirteen base models reliably follow only 1.4-3.6 lines, and extra pretrained loops add little. A task-trained rank-8 LoRA at one early layer extends this computation with all model weights frozen. Qwen3-8B improves from 15.5% to 99% exact accuracy on 24-line chains; a longer-trained LoRA reaches 50 lines. Ouro-1.4B reaches 60 lines after four loops and at least 160 after eight. The LoRA starts a relay: program lines pass on their chain identity through a short range of middle layers. Frozen heads read progressively further up the chain, and removing parent-line attention stops the relay. A frozen-model measurement locates the last useful intervention layer within tolerance in three of four held-out models. Task-specific LoRAs also improve MuSiQue. Default answers therefore understate the computation accessible through a tiny edit. Code and an interactive demo are available at https://lunamos.github.io/stop-thinking-too-early/