AI 学做人时，比「奖励」更关键的是「公平比较」
AI 在开放场景里学互动，有个隐藏陷阱：同一个任务，AI 可以回答、反问、先确认再行动——这些行为本身没有对错，但强化学习里比较不同 AI 的「表现」时，如果它们用的策略不同，奖励模型就会偏好某种风格(比如爱确认的)，而不是看谁更懂上下文。研究者把这个问题叫「奖励公平性」，并给出一个解法：先把 AI 的策略(比如「直接答」还是「先问」)打上标签，再在同类策略里比较，这样奖励才公平。他们用这个方法训练了一个新模型，在工具调用基准上大幅提升，而且首次回复时间从 4.91 秒降到 1.27 秒。这不是你明天能用的技巧，但它揭示了一个关键认知：AI 学互动时，比「给多少奖励」更根本的问题是「比较是否公平」。
📄 原文摘要(英文)
Open-ended real-world interaction admits multiple valid behaviors: an agent may answer directly, ask for clarification, provide progress updates, or confirm before acting. This flexibility breaks a core assumption behind group-based RL: rollouts compared within a group are no longer guaranteed to be behaviorally comparable. As a result, reward-model preferences over interaction style can distort relative advantages and steer optimization toward reward-preferred behaviors rather than context-appropriate ones. We formalize this as a reward fairness problem and propose ARC (Advantage Regularization via Conditioning), a training recipe that restores fairer relative comparison through strategy-conditioned rollout grouping, together with hybrid rewards and entropy regularization. We study ARC in our proposed \inter, a novel paradigm for responsive, steerable, and execution-aware user-agent interaction that decouples user-visible communication from latent reasoning and tool use. \inter\ also provides the annotation and distillation pipeline for constructing \inter-86K, our strategy-annotated training corpus for supervised and RL training. Empirically, ARC substantially strengthens the core τ/τ^2 tool-use benchmarks, while \inter\ reduces time-to-first-token from 4.91s to 1.27s relative to a think-style baseline. Together, these results suggest that a central bottleneck in open-ended interactive learning is not only how agents are rewarded, but whether their behaviors are compared fairly in the first place. The ARC implementation and \inter-86K training data will be released.