视频生成正在从“画”变成“拍”
视频生成不再只是让模型一口气画完一段视频,而是变成了一套“拍摄系统”:先规划脚本、分镜头,再调素材、生成、剪辑,每一步都有工具和反馈在中间协调。研究者把这套协调机制叫“agent harness”,它像导演的现场调度,让模型能拍出连贯的多镜头视频,而不是一段孤立的画面。更关键的是,这套系统还能自己改进自己——从改输出、改参数,到改整个改进流程本身。它不是你明天就能用的工具,但它标志着一个转向:视频生成正在从“画”变成“拍”。
📄 原文摘要(英文)
Video generation is shifting from end-to-end synthesis toward agentic generation, where planning, tool use, and feedback enable coherent, controllable multi-shot videos. In parallel, video world models provide action-conditioned observations and predicted futures, serving as interactive environments for agents. We study their intersection through the agent harness: an executable runtime that constructs context, exposes tools, maintains state, routes feedback, and manages budgets around video models. This report examines three directions. First, agentic video generation, where harnesses coordinate scripting, shot planning, asset management, production, and editing, alongside the evaluation challenges of these systems. Second, agentic world modeling, where harnesses mediate observation, control, execution, and state. We discuss hybrid designs in which game engines or agent-written programs maintain world state while video models render observations, and how agents and world models can mutually evaluate and improve one another. Third, recursive self-improvement (RSI) in video model development, distinguishing output revisions, updates to harness state, generator parameters, and evaluators, and changes to the improvement procedure itself. We identify how these improvements feed back into subsequent development. Across these directions, we analyze essential agent capabilities, including audiovisual perception, long-horizon planning, memory, code-based interaction, feedback-guided control, skill reuse, and multi-agent coordination. We conclude with challenges in autonomous video production, lifelong memory, programmable hybrid control, autonomous model development, provenance, cost accounting, and evaluation linking video quality and task performance to sustained improvements in models and harnesses.