AI终于学会在3D世界里“想”了
现在的AI看图片或视频，其实是在看一张张独立的“照片”，它不知道这些照片里的东西在真实空间里怎么摆、有多大、离你多远。这篇论文让AI像人一样，把多张照片或视频片段拼成一个连续的3D场景，然后在这个场景里“想”问题——比如数一数房间里有几把椅子、判断两个物体谁左谁右、量一量桌子多宽。它不靠重新训练模型，而是给现有AI加了一套“工具箱”：先让AI决定需要什么证据（比如“我要看侧面”），然后用工具在2D图上找到物体，再把它“抬”到3D空间里，最后把多帧的证据攒起来形成结论。这套方法让开源模型在空间推理任务上直接追上甚至超过了GPT-5.4和Gemini 3。它不是你明天就能用的App，但它指明了方向：AI要真正理解物理世界，不能只看“照片”，得学会“搭场景”。
📄 原文摘要(英文)
Real-world spatial intelligence requires reasoning over a continuous and evolving 3D world, yet existing VLMs and tool-augmented agents largely remain tied to static, stateless inference from isolated visual observations. We introduce \textsc{S-Agent}, a spatial tool-use agentic paradigm for understanding and reasoning over continuous multi-view images and videos. By formulating spatial reasoning as spatio-temporal evidence accumulation rather than isolated frame-level prediction, S-Agent reshapes spatial perception into scene-centric understanding beyond frame-centric recognition. Specifically, S-Agent casts the VLM as a semantic planner that decides what evidence is needed, while a hierarchy of spatial tools and experts grounds objects in 2D, lifts them into 3D geometric evidence, and aggregates this evidence into high-level spatial knowledge (e.g., counting, measurement, orientation, and relative position). Additionally, a temporal memory mechanism, including Scene Memory for maintaining the evolving scene state and Agent Memory for accumulating reasoning context, enables evidence integration across frames and reasoning steps. Comprehensive experiments on multi-view and video spatial reasoning benchmarks show that S-Agent consistently improves both open-source and closed-source VLMs in a training-free manner. Beyond inference-time augmentation, supervised fine-tuning (SFT) on S-Agent-generated spatial trajectories S-300K yields S-Agent-8B, a compact spatial agent that significantly surpasses similar-scale baselines (e.g., Qwen3-VL-8B) and performs comparably to advanced closed-source models (e.g., GPT-5.4 and Gemini 3).