AI学会看视频学技能,像人一样边看边做
现在的AI智能体大多靠手写代码或文本指令来学新技能,但人类最自然的学习方式是看视频、读文章、翻资料。这篇论文让AI能像人一样,从教程视频、代码仓库、文章等多媒体资源中自动提炼出可执行的技能,并整理成一本带图、带代码、带视频片段的「技能百科」。在7个实际创作任务中,用了这套方法的智能体平均得分比不用技能的高出11.9个百分点,在28个测试组合中有26个击败了强基线。它不是你明天就能直接用的工具,但指向了一个方向:未来AI可能不再需要你写指令,看一遍你操作就能学会。
📄 原文摘要(英文)
Skills are a useful abstraction for software agents, turning human and agent experience into reusable procedural knowledge. Yet existing skill libraries are mostly hand-written, text-centric, or derived from agent traces, leaving tutorial videos and other multimodal human resources largely underused. We present RESOURCE2SKILL, a framework that distills multimodal resources, including tutorial videos, repositories, articles, and reference artifacts, into executable skills for software agents. RESOURCE2SKILL organizes these skills as a hierarchical multimodal Skill Wiki, where each entry combines structured text, code, visual examples, metadata, and provenance. This design preserves complementary signals from different resources: videos capture temporal operations and visual effects, code captures executable tool patterns, and articles or artifacts provide conceptual and stylistic grounding. At inference time, agents retrieve and compose relevant skills from the wiki; when coverage is insufficient, the same construction operator can acquire new skills online. Across seven practical authoring domains, RESOURCE2SKILL improves average overall score by +11.9 percentage points over no-skill agents and outperforms strong harness baselines in 26 of 28 main-aggregate model-domain cells. Ablations confirm the value of multimodal skill format, hierarchical organization, source diversity, selection strategy, and online acquisition.