AI 操作手机时，终于学会只看该看的东西
现在的 AI 操作手机界面，是把屏幕截图、声音、视频、历史操作一股脑全喂进去，每步都看全部。问题是：上一步的关键信息，下一步可能就成了干扰，错误还会滚雪球，最后 AI 彻底跑偏。这篇提出一个轻量适配器 FocusGUI，让 AI 每一步只挑当下最相关的信息看，把其余压掉，再悄悄塞进模型内部，不改变模型本身。在四个主流模型上，操作准确率平均提升 33 分，精确匹配提升 3 倍。它不是你明天就能用的产品，但这是 AI 从「笨拙地看一切」走向「像人一样抓重点」的关键一步。
📄 原文摘要(英文)
Graphical User Interface (GUI) agents increasingly act over streaming omni-modal observations spanning screenshots, audio, video, interaction history, and device states. However, exposing this complete context uniformly at every step ignores that relevance is transient and sparse: the signal determining one action can become clutter at the next. Because executed actions feed back into the context, these errors compound into Sequential Distraction, driving the agent into irreversible trajectory drift. We formulate the underlying requirement as Focused Perception: a representation that is step-adaptive, action-sufficient, and non-intrusive. We realize this in FocusGUI, a lightweight adapter that dynamically prioritizes multimodal channels against a step-specific decision reference. Instead of uniformly processing all inputs, it compresses the most relevant evidence into a compact latent and subtly injects it into the frozen decoder's embeddings, guiding the model's attention without altering its native parameter space. FocusGUI trains from action supervision alone, requiring neither modality-relevance labels nor backbone fine-tuning. Across four omni-modal backbones, it raises overall Type Match by +33.2 and Exact Match by 3.0 times over the frozen backbone on average, and Type Match by +30.1 over the strongest existing enhancement method.