多轮强化学习只在预算充足时才赢
多轮强化学习被当成让 AI 智能体变强的万能药，但它的优势其实很挑条件：只有在允许 AI 交互足够多轮时，它才明显胜过单轮训练；一旦把交互预算收紧，优势几乎消失。而且它提升的不是每一步决策的质量，而是出错后的恢复能力和长任务中的韧性。研究者还发现，训练时固定用多轮，部署时却可能遇到预算不足，于是提出 BudgetMix RL——训练时混合不同长度的交互轮次，让模型在宽裕和紧张的场景下都稳。它不是你明天就能用上的，但提醒了一件事：别把某个训练方法的效果当成铁律，它可能只是特定预算下的产物。
📄 原文摘要(英文)
While multi-turn agentic reinforcement learning (RL) excels on long-horizon tasks, its performance gains remain highly sensitive to interaction constraints. Evaluating across sampling breadth K and interaction depth T via Pass@(K,T), we show that trajectory-level RL advantages depend on sufficient turn budgets, diminishing significantly when budgets tighten. Besides, agentic RL does not uniformly elevate single-step decision quality; rather, its strength concentrates in error recovery and long-horizon resilience. To mitigate the deployment risks caused by this budget dependence, we provide a preliminary discussion on the benefits and limitations of process supervision signals and introduce BudgetMix RL as an alternative that varies interaction budgets during training. Experiments demonstrate that BudgetMix RL achieves robust performance across diverse execution settings. Project Page: https://rethinking-agentic-rl.github.io/