用代码代替文字描述视频，AI 反而更懂
我们一直默认「描述视频」就该用自然语言。这篇提出反着来：把视频内容写成可执行的代码——用图形元素、布局规则和时间函数去表达画面。结果在问答和视频重建任务上，代码描述比自然语言更准、更保真。但问题来了：现在的多模态大模型还写不好这种代码，尤其复杂的时间动态，开源模型差距更大。这不是你明天能用的技术，但它提示一个方向：也许让 AI 理解世界，不该只逼它说人话。
📄 原文摘要(英文)
Video captioning aims to preserve visual information through compact descriptions, predominantly using natural language. However, precisely expressing fine-grained spatial structures and temporal dynamics in natural language often leads to lengthy and potentially ambiguous descriptions. In this work, we explore Code as Video Caption, a paradigm where executable programs serve as a complementary representation of visual content through graphical primitives, layout rules, and functions of time. To examine whether code captions effectively preserve video information, we compare them with natural language captions on downstream question answering and video reconstruction tasks. Our results show that code captions achieve superior reasoning accuracy and reconstruction fidelity. Despite these advantages, realizing the potential of code captioning depends on whether current multimodal large language models (MLLMs) can generate faithful code captions. To systematically assess this capability, we introduce CodeCap-Bench, which comprises 323 videos spanning five categories of structured 2D content, along with fine-grained annotations and a multi-level evaluation framework. Extensive experiments reveal that while leading proprietary MLLMs demonstrate promising code captioning capabilities, they still struggle to faithfully capture complex temporal dynamics, with open-source models lagging substantially behind.