AI 学多门课，终于知道哪句话该给哪门课打分
现在训练大模型，常常同时给它好几条验收标准：工具调用格式对不对、SQL 查得准不准、指令里的每个约束有没有满足。以前的做法是把每条标准的打分平均摊给整段回答——哪怕这条标准只检查其中几个词，其他词也跟着挨夸或挨骂，学起来又慢又乱。这篇提出 MRTPO:让模型把自己的回答重放一遍，分别带着和不带某条标准的检查信息，对比两种情况下每个位置下一个词的概率变化，变化大的位置就是这条标准真正在意的词，打分只发给它们。在文本转 SQL、多约束指令跟随和工具调用上，它比现有方法收敛更快、效果更好。这不是你明天能直接用的功能，但它指向一个更本质的趋势：AI 的训练正在从「整体给分」走向「逐词问责」，以后模型会更清楚自己哪一步做对了、哪一步做错了。
📄 原文摘要(英文)
Language models are increasingly trained against several verifiable rewards at once, such as the correctness and the format of a tool call, or the individual constraints of a complex instruction. Reward-decoupled normalization (GDPO) normalizes each reward separately so that different reward combinations keep distinct advantages, but it still spreads every reward's advantage evenly over the whole response, although each verifier checks only a few positions. We propose MRTPO, which assigns the advantage of each reward to the tokens relevant to it, using the information its verifier checks, without auxiliary models or token-level annotations. For each reward, the policy replays its own response with and without that reward's verifier information, and the per-token Jensen-Shannon distance between the two next-token distributions measures each token's relevance and reassigns the advantage. On text-to-SQL, multi-constraint instruction following and tool calling, MRTPO converges faster and achieves better performance than the compared methods.