AI视频生成终于能让人和物体“互动”了
现在的AI视频生成，要么只能生成一个人，要么只能生成一个物体，人和物体之间的互动（比如人拿着杯子、穿着带logo的T恤）总是穿帮或模糊。这篇研究把问题拆成两层：一是人和物体之间的“关系”要真实（比如手怎么握杯子），二是物体本身的细节要保真（比如logo不能变形）。他们用多模态大模型（MLLM）来理解“人拿杯子”这种抽象关系，再通过一种新的注意力机制，把这种关系信息注入到视频生成过程中，同时保留对物体细节的控制。结果生成的视频里，人和物体的互动自然，物体细节也清晰。它不是你明天就能用的工具，但方向对了：AI视频生成正在从“单主体”走向“多主体互动”。
📄 原文摘要(英文)
Human-object centric video personalization (HOCVP) is a core task within subject-driven video generation. However, existing methods suffer from two key limitations. First, most approaches focusing on inter-subject personalization still struggle to strike a balance between high subject fidelity and accurate interaction patterns between humans and diverse objects, especially when objects represent abstract concepts such as logos. Second, while intra-subject references (e.g., OCR maps, multi-view inputs) are expected to enhance subject fidelity, most existing works lack mechanisms to understand such latent correspondence. To address both challenges, we propose HOMIE, an HOCVP framework that tackles both inter- and intra-subject input settings in a unified manner. Compared to previous approaches, HOMIE proposes a better MLLM integration strategy to extract knowledge of reference-level relationships without compromising the controllability of text encoders or incurring costly re-alignment. Specifically, we introduce global multimodal guidance within self-attention to better align MLLM-derived semantic features with VAE tokens. Furthermore, we propose modality-reference embedding to differentiate tokens from MLLM features and VAE tokens and associate intra-subject reference image tokens. Extensive experiments validate that our method achieves state-of-the-art performance across various HOCVP tasks. Project Page: https://yiyangcai.github.io/homie-page.github.io/