AI看街景很准，一走路就迷路
多模态大模型看一张街景照片，能认出店铺、路口、距离，单项能力都及格。但把它们放进一个按真实香港三维数据复刻的城市里，让它们自己走、自己找路，问题就全暴露了：方向感不可靠，遇到行人不会避让，更致命的是——走几步错一点，错误不断累积，没有自我纠错机制，最后彻底迷失。研究者造了个叫 UrbanGround 的沙盒，让 AI 以第一人称视角在数字城市里自由探索，专门测试这种“从看懂到走对”的跨越。结果很诚实：局部感知和短距离推理是有的，但一旦目标变远、路线变复杂，这些能力就拼不成持续的目标导向行为。这不是你明天能用上的工具，但它划出了一条重要的分界线：AI 看懂世界，和 AI 在世界上行动，是两回事。
📄 原文摘要(英文)
Multimodal large language models (MLLMs) can interpret a street view, but urban agency depends on whether such local evidence remains useful after the agent starts to move. In this paper, we investigate how far current MLLM agents can turn local urban perception into reliable action in a complicated real-scale city. We propose UrbanGround, the first sandbox to make this question testable in a physically constrained replica of Hong Kong built from territory-wide 3D geospatial data. UrbanGround supports closed-loop interaction from a first-person view and provides an interactive map for navigation. Agents can directly enter the 3D city and explore from a first-person view. Our analysis follows the growth of the spatial problem through three research questions. We first test whether an agent can ground a local scene well enough to answer spatial questions after active observation. Then we ask whether that grounding supports navigation as destinations become farther away and less explicit. Finally, we examine whether the resulting behavior survives changes in route availability and pedestrian motion. Contemporary MLLM agents usually show useful atomic abilities in visual recognition and short-range spatial reasoning, while orientation and pedestrian-aware movement remain unreliable. Their central failure emerges over extended exploration, where local abilities do not compose into sustained goal-directed behavior and errors accumulate without effective correction. We hope UrbanGround will support broader study of how far current MLLM agents can explore reliably in complex, open-ended urban environments.