AI学会在关键时刻“拐弯”了
训练AI用工具（比如查资料、写代码）时，以前的方法只在它调用工具时才给反馈，就像只看考试结果不看解题过程。这篇研究发现，AI真正影响成败的决策点其实遍布在整个思考过程中，很多关键判断发生在它“犹豫”或“突然改变主意”的时刻。他们提出新方法，让AI在这些细小的决策点主动“分岔”探索，并更精准地奖励那些导致好结果的分支。在13个测试中，AI使用工具的成功率平均提升了近4个百分点，而且没有增加不必要的工具调用。虽然你明天用不上，但它让AI变得更像人类——不是只在行动时学习，而是在思考的每一步都学会判断。
📄 原文摘要(英文)
Recent advances in agentic Reinforcement Learning (RL) have substantially improved the multi-turn tool-use capabilities of large language model agents. However, most existing methods assign credit over coarse heuristic units, such as tool-call boundaries or fixed workflows, making it difficult to identify which intermediate decisions influence downstream outcomes. In this work, we study agentic RL from two perspectives: where to branch and how to assign credit after branching. Our pilot analysis shows that influential decision points are broadly distributed throughout the generated sequence rather than concentrated at tool calls, while token entropy alone does not reliably reflect their impact on final outcomes. Motivated by these observations, we propose Agentic Procedural Policy Optimization (APPO), which shifts branching and credit assignment from coarse interaction units to fine-grained decision points in the sequence. APPO selects branching locations using a Branching Score that combines token uncertainty with policy-induced likelihood gains of subsequent continuations, enabling more targeted exploration while filtering out spurious high-entropy positions. It further introduces procedure-level advantage scaling to better distribute credit across branched rollouts. Experiments on 13 benchmarks show that APPO consistently improves strong agentic RL baselines by nearly 4 points, while keeping efficient tool-calls and maintaining behavior interpretability.