AI打游戏有了统一考卷：5000小时真人录像当标准答案
现在的AI评测游戏能力，要么只测几款游戏、要么没有语言指令、要么测试结果随机性太大。这篇论文给AI打游戏立了一套统一考卷：先让100个真人高手玩21款AAA大作、录下5000小时带操作和语言指令的视频，再从中抽出几千道标准题，让47个模型考了一百多万次。结果发现任务难度有明显梯度，模型之间的差距也很大——比如看懂画面容易，但把长指令拆成一步步操作并执行，很多模型就垮了。这套考卷的价值在于：以后比AI游戏能力，大家有了同一把尺子，而且离线考完还能上线复测，看分数是不是真能反映实战。它不是你明天能用上的东西，但它是衡量AI从“看懂”到“会玩”到底差多远的一把标准尺。
📄 原文摘要(英文)
Modern video games provide a measurable testbed for AI models, combining abilities of visual understanding, instruction decomposition, goal planning, and precise action control over multiple temporal horizons. Existing datasets and benchmarks, however, either cover a narrow range of games, lack language instructions, or rely on high-variance online rollouts. To address these challenges, we introduce GameHorizon, a unified data and evaluation suite that measures gameplay capabilities at different horizons for diverse model families. GameHorizon Suite consists of three components. First, GameHorizon-Annotator is a scalable and automated annotation pipeline for multi-horizon instructions. Second, utilizing the pipeline, we construct GameHorizon-Data, the first large-scale AAA gameplay dataset with temporally aligned videos, player actions, and multi-horizon instructions. It comprises 5,000 hours of recordings from 21 games, collected by 100 human expert players. Third, we build GameHorizon-Bench with reproducible offline and stepwise online testing. The offline track enables reproducible evaluation using thousands of standardized questions organized into three primary tasks and a series of diagnostic variants, while the online track tests whether offline scores reflect actual gameplay capabilities and localizes failures to specific steps within long-horizon gameplay. Based on our GameHorizon Suite, we evaluate 47 models through more than one million model invocations, revealing a meaningful hierarchy of task difficulty and pronounced differences in model capabilities. Our work can provide a standardized yardstick for evaluating gameplay capabilities across horizons and model families. We will release our dataset, annotator, and benchmark to facilitate future research.