AI Pulse
📄 论文解读

AI 变强了,但思路变窄了:问题出在第一步

强化学习让 AI 答对率大涨,代价是它不再尝试别的路。这篇论文把推理过程拆开看,发现思路的消失几乎全发生在第一步:AI 在写下第一个数字和运算符之前,就已经把其他解法全排除了。证据很硬:第一步之前的概率变化比后续推理大 11 到 16 倍;只要人为补上那个被跳过的开头,完成率能从 1.8% 涨到 21%。也就是说,它不是不会算,是压根不进门。研究者还发现,把早期检查点的参数混进后期模型,能恢复 37% 的解法覆盖,且不损失准确率。这不是你明天能用上的技巧,但它解释了一个正在发生的趋势:AI 越练越专,专到忘了还有别的活法。

📄 原文摘要(英文)

Reinforcement learning with verifiable rewards (RLVR) substantially improves single-sample accuracy (pass@1) but causes the policy's solution space to contract, diminishing the returns of test-time scaling. In this work, we investigate where inside a reasoning trajectory this breadth is lost: does the policy fail to access a valid solution family, or does it fail to execute computation once initiated? To disentangle access from execution, we analyze the Countdown task, whose solution space can be exhaustively enumerated into discrete entrance families defined by the first operand and operator, across PPO on Qwen2.5-3B and GRPO on Qwen2.5-3B-Instruct. Across both training setups, solution coverage falls by up to 67%, halving even on problems solved across all checkpoints. We show that this contraction is heavily concentrated at the entrance: per-token likelihood shifts are 11x--16x larger prior to the first arithmetic operation than during downstream reasoning. Supplying only an unselected entrance prefix restores completion rates in low-access families by over an order of magnitude (0.018 -> 0.212 under PPO), demonstrating that alternative solutions remain executable but are no longer initiated. Guided by this localization, we find that while surface prompting fails to recover diversity, entrance-targeted interventions succeed: late-layer parameter interpolation with early checkpoints increases solution coverage by 37% at no loss in pass@1. Finally, we show that early-step entropy collapse recurs across six math benchmarks with 7B and 14B models, but is not an inevitable byproduct of reasoning optimization: an SFT baseline preserves more than double the coverage, and staged SFT--DPO--RLVR pipelines retain early-step entropy. In summary, reasoning breadth is lost at the door, not inside the room. Code: https://github.com/ershiyidian/early-branch-locking.

arXiv 原文

订阅 AI Pulse

每天 08:00 · 12:30 · 18:30 · 23:50 更新