AI操作电脑：不看屏幕，直接读代码
现在的AI操作电脑，都是先截图、再识别、再点击——就像人一样。但截图会丢失信息：两个不同的程序状态可能显示同一张图。这篇论文反其道而行之：让AI直接读取程序底层状态（文件、DOM、后端数据），用代码操作，只在必要时才用截图点击（108个任务中仅28个需要，且只占主智能体1.1%的步骤）。结果：在OSWorld 2.0测试中，成功率从20.6%提升到26.9%，成本却降到原来的1/9。更关键的是，它揭示了瓶颈从“看”转移到了“想”——失败更多是因为推理错误，而非感知错误。这不是你明天能用的工具，但它指向一个趋势：未来的电脑操作AI可能不再需要“看”屏幕。
📄 原文摘要(英文)
Computer-use agents are usually improved by strengthening perception: better models for reading a screenshot and choosing where to click. Yet a screenshot is only a lossy rendering of the underlying program state, e.g., the files, application backends, and DOM that hold the task data. Different states can produce the same pixels, while code can inspect and modify that state directly. StateAct is a code-first, multi-agent harness built around this distinction. Its main agent works directly with program state by using code, while a dedicated GUI subagent handles screenshot-and-click interaction on the few subgoals that need it, just 28 of 108 tasks and 1.1% of main-agent steps. The same direct access to program state also supports verification: an independent finish gate double-checks the saved result for structural failures, e.g., output that is missing, unsaved, or written to the wrong path. To stay on track over hundreds of steps, the main agent hands subgoals to fresh subagents, keeping its own context focused. On OSWorld 2.0, StateAct lifts Claude Opus 4.8 from 20.6% to 26.9% on binary success, and from 54.8% to 61.6% on partial success, at ~ 9x lower cost per task than the same model driven by screenshots alone; a code-only variant with no GUI subagent reaches only 45.9% partial, below that screenshot-based baseline's 54.8%. In general, grounding action, verification, and memory in state, what we call state-grounding, shifts the main bottleneck from perception toward reasoning: failures depend more on what the agent thinks than on what it sees.