AI终于能同时看懂图片和视频了
现在的多模态AI通常把图片和视频分开处理，就像用两套不同的眼睛。HYDRA-X首次用一个统一的视觉编码器同时处理图片和视频，而且效果更好。它发现：处理视频时，只看相邻帧之间的变化就够了，不需要看所有帧的全貌，这样反而更高效；同时，它用分层压缩的方式，先压缩再解压，让模型既能理解画面内容，又能生成新画面。更关键的是，它改进了图像编辑的流程：让AI在理解画面的早期阶段就进行交互，而不是等到后期用语言模型去猜，这样编辑更一致、训练更快。这个模型在7B参数规模下，图片和视频的理解与生成任务都表现不错。虽然它不是你明天就能用的工具，但它指明了未来多模态AI的发展方向——统一、高效、更懂视觉。
📄 原文摘要(英文)
Holistic visual tokenizers are fundamental to unified multimodal models (UMMs) as they map diverse visual inputs into a unified representation space. In this paper, we present HYDRA-X, the first UMM that unifies image and video tokenization within a single Vision Transformer (ViT). Our design is driven by two core challenges: efficiently injecting spatiotemporal reconstruction capability into a native ViT, and embedding image- and video-level semantic awareness into the latent space. To address the first, comprehensive ablations reveal two key findings: (1) frame-level causal temporal attention suffices for visual reconstruction, whereas full spatiotemporal attention degrades it; and (2) hierarchical temporal compression substantially outperforms single-step alternatives. To tackle the second, we propose a lightweight decompressor that upsamples temporally compressed features under joint image-video teacher supervision, thereby enforcing complementary semantic structures within the compact latent space. Building on this holistic tokenizer, we further propose a principled improvement of the editing pipeline: source-target interaction should occur at the latent level inside the tokenizer rather than at the semantic level inside the LLM, substantially improving editing consistency and accelerating convergence. Instantiated at the 7B dense model, HYDRA-X achieves strong performance across image and video understanding and generation tasks, paving the way for future unified-tokenizer UMMs.