让AI在整座城市里找东西,不再靠猜
过去让AI在3D城市里定位一个目标,得先把整座城切成无数候选块,再挨个打分排序——又慢又笨,还容易漏。这篇换了个思路:把找东西变成一场主动搜索。AI像人一样,先看全景锁定地标,再放大局部,换个角度观察,最后才圈出目标框。研究者用强化学习训练这个搜索策略,在CityRefer基准上拿到了最好的命中率和定位精度,而且不需要预先切块,速度比最强的旧方法快得多。它不是你明天能用上的东西,但方向很明确:AI理解空间的方式,正在从“背答案”变成“会找路”。
📄 原文摘要(英文)
City-scale 3D grounding is commonly formulated as candidate ranking, which requires precomputed instance proposals and becomes inefficient in large outdoor scenes. We propose Urbex, an agentic vision-language framework that instead casts grounding as active spatial search. Urbex represents a city-scale 3D scene as an interactive multi-view environment, allowing a VLM agent to query landmarks, zoom into local regions, render oblique views, and commit a grounded bounding box. We optimize this tool-use policy with reinforcement learning on CityRefer, using smooth localization rewards and lightweight shaping rewards for efficient exploration. Experiments show that Urbex achieves the best Hit@1 and IoU on CityRefer, supports candidate-free grounding without ground-truth boxes, and is substantially faster than the strongest city-scale ranking baseline. Zero-shot results on STPLS3D-Refer further indicate improved cross-scene robustness.