视频生成卡在渲染?不,卡在理解
视频生成早就过了「画得清」的坎,现在卡在「听不听得懂人话」。快手 Kling 团队把自家模型拆开看,发现瓶颈不在渲染,而在从文字到画面的整个理解链条:他们用「语义合同」式的密集描述当中间语言,让模型把每个词都兑现成画面;又发现决定文字编码器好坏的,不是语言能力强,而是它懂不懂画面里的感知细节;连打分模型都分成了「讲道理」和「快决策」两套。这不是你明天能用的功能,但它解释了为什么你让 AI 改个细节它总听不懂——问题不在嘴,在耳朵。
📄 原文摘要(英文)
Video generation has progressed from isolated clips to coherent short-form audiovisual sequences, and the bottleneck is shifting from rendering to semantic control. This report summarizes the observations and insights on multimodal understanding gained from successive generations of Kling. We argue that generation quality is the projection of multimodal understanding through the entire data--conditioning--alignment--evaluation chain, and we examine this chain module by module. Kling-MLLM, a shared multimodal foundation model that unifies image, video, audio, synchronized audio--video, and interleaved inputs, is specialized for each downstream module. Because a DiT is trained on far fewer training samples than a language model, we treat captioning as a semantic contract that maximizes description density, and extend it to reference-grounded captioning that makes reference bindings explicit. Prompt enhancement translates user intent into this caption distribution: basic PE is derived from captions of compelling videos and refined with text-level and generator-in-the-loop reinforcement learning, while agentic PE refines the story and retrieves compelling examples. We show that a reliable text encoder comes from perceptual training rather than from stronger language ability, and that candidates can be screened from text alone before end-to-end training. Reward modeling pairs a general reward model for explainable, localized critiques with a decision reward model for fast multidimensional scoring, treating perceptual bandwidth as a first-class design parameter. Finally, we review task-specific automatic evaluation, focusing on protocols that expose scoring shortcuts and help trace failures to their upstream source. We hope these lessons offer practical guidance for building the understanding stack behind controllable video generation.