AI看视频还是不会用工具
现在的多模态AI看视频已经很能说了——描述画面、回答“发生了什么”都行。但真让它理解“人怎么用工具干活”，它立刻露馅：锤子怎么握、螺丝刀该往哪拧、拧完东西变成什么样，这些它都抓不住。研究者做了第一套专门考“工具使用”的视频数据集和考题：100小时第一视角录像，配上音频、详细解说和3D信息；考题分四类，从“看见工具”到“推理因果”。结果最强模型Gemini-3.1-Pro总分66.9%，但最基础的“感知与定位”只有51.7%——连工具在哪都找不准。这不是你明天能用上的东西，但它划了一条线：AI对世界的理解还停在“看热闹”，没到“看门道”。
📄 原文摘要(英文)
Real-world embodied tasks, from everyday activities to professional procedures, require agents to act under physical constraints while tracking evolving object and task states. Tool use sits at the heart of such tasks, as many everyday and professional activities are tool-mediated. Understanding them requires reasoning about affordances, hand-tool-object geometry, procedural progress, and causal effects on target objects. Yet despite strong performance on perception-oriented video tasks such as captioning and general video QA, current multimodal video models remain limited in this form of tool-centric embodied reasoning. Progress in this direction has been limited by the lack of real-world egocentric data and diagnostic benchmarks. To address this gap, we introduce EgoTools, the first comprehensive suite for egocentric tool-use understanding. It consists of two complementary components: EgoTools-Data, a large-scale corpus of 100 hours of tool-centric egocentric recordings with synchronized audio, dense captions, reasoning-heavy narrations, and supplementary 3D information; and EgoTools-Bench, a diagnostic benchmark of 1,000 QA pairs across four tracks that cover tool-use understanding from perception and geometry to procedure and causal reasoning. Experimental results show that current models still struggle to ground tool use in visual evidence: Gemini-3.1-Pro achieves 66.9% overall accuracy but only 51.7% on Perception & Grounding. Beyond evaluation, we validate EgoTools-Data as a training resource. On the full 1,000-question benchmark, full supervised fine-tuning improves Qwen3-VL-8B-Instruct from 50.0% to 60.9%, under strict source-video separation. Together, these results establish EgoTools as a unified resource for both training and diagnostic evaluation of real-world egocentric tool-use understanding.