AI 学推理，错一次就该停
现在训练 AI 推理，通常只看最终答案对不对，中间过程错了也不管，导致它绕远路、学得慢。这篇提出一个简单办法：让一个现成的 AI 当老师，盯住推理过程中第一次出错的地方，把这次尝试切成「对的前半段」和「错的后半段」，然后只奖励前半段、惩罚后半段。在 12 个任务上，这个做法比主流方法平均提升 7% 到 15%，而且老师 AI 不用很强。它不是你明天能用上的功能，但说明一个趋势：AI 训练正在从「只看结果」转向「细抠过程」，就像老师批改作业时圈出第一个错，而不是只给个分数。
📄 原文摘要(英文)
Reinforcement learning with verifiable rewards (RLVR) has emerged as a powerful paradigm for large language model (LLM) post-training, but its reliance on coarse outcome rewards leads to limited guidance on intermediate reasoning processes. Existing approaches such as process reward modeling and on-policy distillation introduce additional constraints, such as reliance on a specialized reward model or assuming identical reasoning patterns between teacher and student. Nevertheless, we observe that once a reasoning process first goes wrong, evaluating the subsequent reasoning provides limited additional information, as it is already conditioned on an invalid prefix. Therefore, we propose Cliff, a reward shaping strategy that utilizes an off-the-shelf LLM as a teacher to identify the first mistake in each rollout. As a result, the rollout is naturally decomposed into two parts: a correct prefix and an incorrect suffix. Cliff then converts this signal into token-level advantages, assigning positive advantages for the correct prefix and negative feedback afterward. Experiments across 12 different scenarios demonstrate that Cliff consistently improves reasoning performance, outperforming on-policy distillation by 15% and standard GRPO by 7%, even with teachers of modest capability. Furthermore, we analyse the role of ``ground truth'' in Cliff and investigate its training dynamics. These results establish Cliff as a simple, general and effective approach for improving RLVR with richer, fine-grained supervision.