AI 学会像人一样用鼠标键盘打游戏了
我们总以为 AI 打游戏靠的是读内存、开外挂；这篇让 AI 像真人一样，只用看屏幕、动鼠标键盘，在《我的世界》里从零开始合成铁镐。它分两层：上层用视觉语言模型做长期规划、记进度，下层一个 VLA 模型负责具体操作，还专门处理了「点菜单」和「移动视角」这两种动作互相干扰的问题。在标准测试里，它的单步操作成功率 85%，长任务成功率 37.4%，都是目前最好。它不是你明天能用上的东西，但这是 AI 从「会下棋」走向「会玩开放世界游戏」的关键一步。
📄 原文摘要(英文)
Building virtual embodied agents that operate in open-ended games like human players is a long-standing goal requiring both long-horizon autonomy and reliable native-interface control. As a procedurally generated open world, Minecraft provides a natural testbed for studying the integration of these capabilities. Existing agents typically provide either native visuomotor control without long-horizon autonomy or high-level planning executed through environment APIs or predefined skills. To bridge this gap, we propose HiSTEVE, a hierarchical VLA agent with a capability-aware agent harness. At the orchestration level, we equip the harness with structured memory for tracking progress and a VLM planner for selecting subtasks within the VLA's capabilities. To reliably determine subtask completion, we further introduce a policy-coupled task completion state detector that conditions on VLA policy context to signal candidate completion. At the control level, a single VLA executes all subtasks with vision-only perception and native keyboard-and-mouse actions. To mitigate interference between heterogeneous GUI and non-GUI interactions, we introduce regime-specific RMSNorm branches and separate mouse-movement token embeddings. We also train the VLA through world knowledge injection, imitation learning, and reinforcement learning with environment feedback to improve reliability and recovery. HiSTEVE achieves state-of-the-art atomic-task performance, scoring 85.0% on MineStudio Bench and a 37.4% average success rate on OpenHA, while also completing long-horizon tasks such as crafting an iron pickaxe from scratch like human players.