AI边看边教，最强也只过一半
我们评测 AI 助手时，通常让它看一段录好的视频、答一道题，答对就算过。可现实里你边做饭边问它，它的回答会改变你下一步怎么做，你的手忙脚乱又会改变它接下来看到什么——这个循环，离线评测根本测不出来。研究者干脆把「任务环境」和「人」一起模拟：搭出做饭、整理家务、手机操作三个日常场景，再配一个带性格设定的虚拟用户，它会照着 AI 的指导动手，还会故意犯点执行错误、临时改主意。315 个场景跑下来，最强的 Gemini 4 Argon 也只在限时内完成 53.4% 的任务，事实性依据和「能照着做」的指导都明显偏弱；把响应延迟算进去，它常常慢到来不及阻止错误。团队还做了个小实验：让 Qwen3.5-4B 在这个闭环里训练，做饭场景的任务成功率和互动质量都涨了。它不是你明天能用上的工具，但它指出了一个你迟早会撞上的事实——AI 助手好不好，不该只看它答得对不对，得看它带着你做事时，你能不能做成。
📄 原文摘要(英文)
Visual interaction models that support users during ongoing physical tasks require evaluation in closed loop, where assistant responses can change subsequent user behavior and task outcomes. Existing benchmarks largely rely on offline replay and therefore cannot capture these effects. We introduce Interactive Visual Gym, a framework for closed-loop evaluation and training of visual interaction models through joint simulation of executable task environments and human users. We contribute three simulators that recreate everyday tasks and human-aligned action spaces across cooking, household manipulation, and mobile-interface domains, together with a persona-conditioned user simulation harness that collaborates with the assistant while incorporating controlled execution errors and other out-of-plan behaviors. We instantiate the framework as IVG-Bench, a benchmark with 315 interaction scenarios. Evaluating state-of-the-art VLMs as visual interaction models reveals substantial limitations: the best-performing model, Gemini 4 Argon, completes only 53.4% of tasks within the time limit, exhibits significant weaknesses in factual grounding and actionable guidance, and is often too slow to prevent errors when response latency is considered. We verify our evaluation protocol with a human study which finds 78.5% agreement between our interaction-quality metric and participant preferences. We further show that the environments provide useful learning signals: in a proof-of-concept GRPO experiment, closed-loop training improves both task success and interaction quality for Qwen3.5-4B model in our cooking environment.