AI 拍视频不会用工具,现在学会了
现在的 AI 视频生成器看着唬人,但一遇到「按步骤做菜」「保持同一个人长相」「物理上别穿帮」这类要求就露馅。这篇让 AI 学会自己调用外部工具:先拆解你的指令,中途检查生成结果,不对就重来,像人一样边做边改。在 600 条刁钻指令的测试里,它比原版生成器高了 19 分,而且换更强的生成工具还能继续涨分,不用重新训练。它不是你明天就能用的产品,但这是 AI 视频从「会生成」走向「会干活」的关键一步。
📄 原文摘要(英文)
Recent advances in video generative models have enabled high-fidelity, temporally coherent video generation. However, these models often struggle to satisfy prompts requiring specialized knowledge, specific identities, physical consistency, or ordered events. In this paper, we present VideoGen-Agent, a multimodal agent trained through multitask agentic reinforcement learning to use external tools for video generation. The agent coordinates augmentation, generation, and verification tools through multi-turn interactions, using the prompt and intermediate observations to guide its decisions. We train a shared policy on a category-balanced dataset spanning six tasks. Supervised fine-tuning on teacher-generated trajectories establishes tool-use behavior, which is then refined through reinforcement learning. A category-aware hybrid reward evaluates tool-call validity, task-appropriate tool use, and generated video quality. We further introduce VABench, a held-out benchmark of 600 prompts covering procedural knowledge, single- and multi-entity identity preservation, physical consistency, scene composition, and multi-shot temporal structure. On VABench, VideoGen-Agent improves over its base text-to-video generator by 19.1 points, from 56.5 to 75.6. Upgrading the generation tools further raises the score to 86.1 without additional agent training. Human raters prefer the upgraded configuration over the strongest standalone baseline in 84.3% of comparisons. These results support learning tool use across video-generation tasks and show that the trained agent can benefit from subsequent advances in generation tools.