AI 打游戏终于学会“想”了：先预测画面再动手
以前的游戏 AI 是“看见就按”，看到什么画面就直接映射成按键，它不预测下一秒会发生什么。这篇把两件事合并成一个模型：一边生成未来几帧的画面，一边生成对应的键盘鼠标操作，而且两者是同步推演的——它先“想象”画面会怎么变，再决定怎么按。更关键的是，它不只会打游戏，还能操作图形界面(GUI)，并且能自己判断当前该用游戏操作还是界面操作。测试里它用更少的按键次数就完成了任务。研究者还发现一个有意思的失败模式：生成动作时，采样来源里的低频成分会悄悄主导镜头转向，导致动作对采样方式异常敏感。这不是你明天能拿来用的东西，但它指向一个方向：AI 玩游戏的下一步，不是更快地反应，而是先学会“预演”。
📄 原文摘要(英文)
Modern video games combine first-person perception, rapid visual changes, persistent world state, and heterogeneous native controls. Existing game agents map visual and task context directly to actions but lack explicit world dynamics modeling, whereas interactive game world models predict visual futures from supplied actions but do not serve as task policies. World-Action Models (WAMs) unify these objectives, but remain largely unexplored under the dynamics and open-ended interaction of video games. We introduce GameWAM, to our knowledge the first WAM for native closed-loop gameplay and GUI control. GameWAM jointly generates future visual observations and executable keyboard-mouse trajectories through parallel visual and action generative processes with block-causal conditioning and flow matching. To support joint world-action learning, we construct synchronized gameplay and GUI trajectories. To handle heterogeneous native control, GameWAM predicts a gameplay/GUI mode at each action step and generates actions with mode-specific prediction distributions and continuous-action normalization. For long-horizon interaction, block-cycle control predicts beyond the committed horizon, executes only a short action prefix, and replans from new observations, while fine-grained within-cycle context and hierarchical cross-cycle history preserve temporal continuity. Experiments demonstrate competitive task success with fewer executed native actions than the compared agents. We further uncover Low-Frequency Action Source Imprinting (LASI), in which low-frequency components of the sampled action source systematically steer coarse generated camera motion under fixed conditioning, revealing a source-sensitivity failure mode in generative control. Project page is available at https://yunncheng.github.io/GameWAM/.