训练AI时多监督一个东西，探索能力反而更强
训练AI智能体时，标准做法只让模型预测自己会做什么动作，环境反馈只是背景信息。这篇论文发现，如果同时让模型预测环境接下来会发生什么——即使它实际运行时根本不需要预测——模型在后续强化学习中的探索能力反而更强。在多个基准上，这种训练方式让模型在多次尝试中解决更多不同任务，而且最终策略更接近初始状态、保留更多随机性。原理是：只监督动作会让模型逐渐丧失对后果的预测能力，而同时监督观察能防止这种偏科。这不是你明天能直接用的技巧，但它提示了一个反直觉的方向：让AI学习时多关注它不需要输出的信息，反而能帮它更好地理解世界。
📄 原文摘要(英文)
Agent trajectories record what an agent does and what happens next. Yet standard supervised fine-tuning (SFT) applies loss only to agent-authored action tokens, using environment observations as context but not as prediction targets. We ask whether this convention provides the best initialization for subsequent reinforcement learning. We introduce ActObs, which also supervises the observation tokens already present in each trajectory. Although deployed agents never generate observations, learning to predict them encourages the policy to model action consequences without adding data, parameters, sequence tokens, or forward passes. The methods perform similarly after SFT but diverge after GRPO. On Qwen3-4B, GRPO from ActObs achieves higher pass@k at every evaluated sampling budget than its action-only counterpart on Terminal-Bench 2.0. On Qwen3-8B, it trades some pass@1 reliability for higher pass@k (+3.4 pp at pass@16) and solves more distinct tasks. The advantage extends to cross-domain code editing on aider-polyglot (+4.2 pp at pass@1 at 4B), whose tasks are unseen during SFT and RL. ActObs retains more entropy during RL while requiring less policy movement, leaving the final policy closer to its SFT initialization. Our analysis traces this difference to SFT: action and observation gradients rapidly become orthogonal, while action-only training leaves a large residual observation gradient and degrades environment prediction below the base model. Joint supervision prevents this one-sided specialization, preserving consequence prediction and preparing the policy for downstream exploration.