AI 干活时，你中途改主意，它就跟不上了
现在的 AI 助手被训练成「一次听清指令、然后闷头干完」，但真实工作里人总会中途改需求、纠正错误、追加限制。这篇论文造了一个测试：让 AI 在命令行环境里干活，同时安排一个「脚本用户」不断插话——改条件、纠错、收回权限。结果最强配置平均分只有 49.2%，而把「协作质量」单独算分时，最高才 37.6%。也就是说，AI 能完成任务和能配合你，是两码事；它尤其栽在「你让它停它就停」「你信它几分」「它敢不敢越界」这三件事上。这不是你明天能用的工具，但它指出了一个真问题：我们总以为 AI 越强越省心，其实它越强，越需要你盯着它别跑偏。
📄 原文摘要(英文)
Command-line interface (CLI) agents increasingly execute longer-horizon tasks, yet existing benchmarks either assume that user instructions remain fixed throughout execution or evaluate task outcomes without capturing runtime collaboration quality. In this paper, we present REALCLI, a paired benchmark that unifies the evaluation of task execution and human-agent collaboration quality on realistic CLI tasks. To comprehensively measure collaboration, REALCLI builds a five-dimensional framework, operationalized into 22 sub-dimensions and 51 interaction scenario templates. REALCLI provides two modes for paired comparison: REALCLI-EXEC supplies complete specifications upfront, whereas REALCLI-COLLAB introduces a scripted user that delivers progressive information, corrections, constraint revisions, and authorization decisions. Evaluations reveal profound capability gaps in frontier agents: the best configuration attains an average score of only 49.2%, and when evaluated by the joint collaboration score (J_{collab}), the peak performance reaches merely 37.6%. These results show that task effectiveness diverges sharply from collaboration quality, with models consistently struggling in steerability, trust calibration, and safety boundary management. REALCLI provides a rigorous, reproducible foundation for measuring and advancing collaborative CLI agents.