AI终于能记住多人对话里谁对谁说了什么
现在的AI记忆只擅长一对一聊天，一旦饭桌上三个人同时说话，它就分不清谁在跟谁讲、谁说过什么。这篇论文做了一个叫VoxPolyMem的系统，专门解决多人语音对话的长期记忆：它一边听一边给每个说话人建档案，把对话拆成三层——谁说了什么、事实是什么、这个人是谁——需要回忆时，AI会像侦探一样自己决定先查哪层、再问什么，而不是把所有内容一股脑塞进去。在专门为多人对话设计的测试上，它拿了85分，比最强的现有方案高出23.6分。它不是你明天就能用的产品，但这是AI从“记住你一个人”走向“记住你们一群人”的关键一步。
📄 原文摘要(英文)
Long-term memory enables agents to accumulate information and reason across sessions, yet existing research primarily focuses on dyadic text or image-text conversations, leaving long-term memory for multi-party spoken conversations underexplored. This setting requires preserving conversational content, identifying participants across sessions, and retaining who speaks to whom. To this end, we propose VoxPolyMem, an interaction-aware multimodal memory framework combining incremental speaker identification with a memory hierarchy comprising interaction memory, fact memory, and participant profiles. We formulate retrieval as sequential decision-making, where an agent rewrites queries and selects retrieval tools and memory layers based on accumulated evidence to address information gaps. We further introduce Evidence-Gain GRPO (EG-GRPO), which uses round-wise credit assignment to encourage complementary evidence acquisition. We also construct VoxPolyBench to evaluate memory evolution, personalized answering, memory retrieval and reasoning, and interaction reasoning and attribution in multi-party spoken conversations. VoxPolyMem achieves an overall score of 85.0 on VoxPolyBench, surpassing the strongest evaluated baseline by 23.6 points. On Mem-Gallery and H2HMem-Multi, it scores 89.6 and 74.4, respectively, exceeding the strongest evaluated public memory baselines by over 8 points each. These results highlight its potential for persistent, personalized assistance in multi-party multimodal interactions. Code and datasets are available at https://voxpolymem.github.io/VoxPolyBench/demo/