AI学技能不再靠外挂，自己复盘就能变强
训练AI智能体（比如帮你订机票的对话机器人）时，一个核心难题是：它做完一整件事后，只知道最后成没成，但中间哪一步对、哪一步错完全没反馈。这篇论文让AI自己从做过的任务里提取“技能”——比如“先查库存再下单”这种全局规则，或者“用户说“便宜点”时立刻推荐折扣款”这种局部技巧。它不依赖外部知识库，而是从当前策略刚跑完的轨迹中实时提炼，确保技能和当前状态匹配。在三个测试环境（家务、购物、问答）中，这种自我复盘式训练让AI学得更快、更稳，尤其在复杂多轮对话中表现更鲁棒。它不是你明天就能用的工具，但揭示了强化学习训练语言模型的一个新方向：让AI自己当自己的教练。
📄 原文摘要(英文)
Outcome-based reinforcement learning provides a stable optimization backbone for language agents, but its sparse trajectory-level rewards provide little guidance on which intermediate decisions should be reinforced or suppressed. On-policy self-distillation offers dense token-level supervision, yet existing skill-conditioned variants often rely on external skill memories or retrieved privileged context, which are costly to maintain and can be mismatched with the state distribution induced by the current policy in multi-turn interaction. We propose OPID (On-Policy Skill Distillation), a framework that extracts skill supervision directly from completed on-policy trajectories. OPID represents trajectory hindsight as hierarchical skills: episode-level skills capture global workflows or failure-avoidance rules, while step-level skills capture local decision knowledge at critical timesteps. A critical-first routing mechanism uses step-level skills when critical decisions are identified and falls back to episode-level skills as default guidance otherwise. The selected skill is injected into the interaction history, allowing the old policy to re-score the same sampled response under both original and skill-augmented contexts. The resulting log-probability shift yields a token-level self-distillation advantage, which is combined with the outcome advantage for policy optimization. OPID thus preserves RL as the primary training objective while introducing dense, distribution-matched hindsight supervision. Experiments on ALFWorld, WebShop and Search-based QA demonstrate that OPID generally improves agent performance, sample efficiency, and robustness over outcome-only RL and existing skill-distillation baselines. Our code is available at https://github.com/jinyangwu/OPID/tree/main.