视频理解开源模型终于不瘸腿了
开源视频AI模型一直有个尴尬：要么只能看懂特定类型视频，要么跑起来慢得离谱，而且代码、数据、训练方法总藏着掖着。这篇直接把所有东西全公开，还解决了两个核心痛点：一是用「膨胀3D视觉Transformer」把视频处理成本砍到接近图片级别，二是搞了一套自动生成训练数据的流水线，让模型在短视频、长电影、直播流三种场景下都能打。4B参数就干翻更大模型，跑得还更快。它不是你明天就能用的工具，但如果你在等一个真正能拿来搞视频分析的开源基座，这就是信号。
📄 原文摘要(英文)
Recent advances in video understanding have spanned motion, long video, and streaming interaction, driving this field toward real-world applications. Despite this progress, current open-source models remain limited in several ways. They often struggle to generalize across diverse video types, making them effective only in specific domains. High computational demands further restrict their efficiency and scalability. Moreover, most models are only partially open, with key components such as training code, strategy, or datasets unavailable, which hinders reproducibility and slows community-driven development. To address these issues, we introduce VideoChat3, a fully open, efficient, and generalist video-centric MLLM. VideoChat3 advances video understanding through two complementary designs. For efficiency, we introduce Inflated 3D Vision Transformer (I3D-ViT) and Adaptive Frame Resolution for Streaming Video Perception, which enables efficient spatiotemporal representation and reduces the cost of processing video inputs during training and inference. For effectiveness, we develop a scalable video data synthesis pipeline that curates three diverse, high-quality training datasets: VideoChat3-Academic2M, VideoChat3-LV116K, and VideoChat3-OL617K, covering general, long-form, and streaming video scenarios, improving the model's generalization across domains. By integrating these designs, VideoChat3 achieves a rare balance of broad generalization and computational efficiency. Experiments across general, long-form, and streaming benchmarks demonstrate that VideoChat3 surpasses prior open-source models with equal or larger parameter counts with only 4B parameters and higher efficiency.