AI Pulse
📄 论文解读

不教模型认路,它反而更会走

让 AI 在陌生房子里找路,常规做法是把负责「看懂画面」的大模型按导航数据再训一遍——代价是它原本的通用视觉本事被磨掉。这篇反着来:大脑不学「往左三米」这种坐标,只学会用自然语言喊话,比如「绕过沙发往前走」,由另一个专门的动作模块把话翻译成连续移动;两者再用闭环强化学习一起调,大脑的权重几乎没动。在标准连续环境导航测试里,它明显赢过那些被专门微调过的分层方法,同时基本保住了通用视觉能力。它不是你明天能用上的东西,但指向一个趋势:让大模型当「下指令的人」而不是「被改写的执行者」,可能是保住通用能力又拿下专用任务的一条路。

📄 原文摘要(英文)

Representative vision-language navigation (VLN) methods typically couple a vision-language model (VLM) brain with a learnable action head and fine-tune the VLM brain on navigation-domain data to predict geometric goals. However, such adaptation can substantially alter base VLM weights and degrade general visual capabilities. We introduce ZeroVLN, a slow-fast navigation framework that achieves effective navigation with zero navigation-domain SFT of the VLM brain. Rather than teaching the brain to predict geometric goals, we equip it with a semantic tool-calling interface for conveying movement decisions to an action expert in natural language. We first adapt the action expert to carry out these movement requests, then jointly improve the brain's movement decisions and the action expert's continuous execution through closed-loop reinforcement learning (RL). Trained on the same navigation episodes, ZeroVLN substantially outperforms representative SFT-based hierarchical VLN methods on standard VLN-CE (continuous environments) benchmarks, while inducing only minor changes to the brain's weights and largely preserving its general capabilities. To our knowledge, ZeroVLN is the first slow-fast VLN framework with zero navigation-domain SFT of the VLM brain, and the first to jointly optimize its slow and fast policies through closed-loop agentic RL in VLN-CE.

arXiv 原文

订阅 AI Pulse

每天 08:00 · 12:30 · 18:30 · 23:50 更新