让智能体记住用户模样声音，不只是记住他说过的话。
Parametric Multimodal User Memory: Storing What Captions Cannot Carry
- 用视觉语言模型定位上下文，专用编码器提取身份特征
- 联合使用时识别准确率达0.96，支持多说话人音视频
- 无需训练，注册成本为O(1)，适合个性化智能体
个性化智能体需要持久的用户记忆：对用户身份的持续建模。当前几乎全依赖文本——通过相似性检索的对话转录和字幕。这仅覆盖可被描述的部分（如“我的猫叫Bibi”），却丢失了无法用文字表达的感知信息：声音特征、面部在不同年龄与光照下的变化、语气中的疲惫感。我们在五个模态上评估此损失，发现基于字幕的重识别器对非命名信号的召回率低至0.11，趋近随机水平。为此，我们提出将感知记忆嵌入模型内部，将识别分解为两个子问题：视觉-语言模型在上下文中定位指代对象（是什么、在哪里），专用编码器提取身份密钥（是谁），以一个内联令牌形式存储，生成时由注意力机制直接读取，无需外部查询。单独使用时效果有限：视觉语言模型在跨年龄人脸识别中召回率仅0.54，而专用编码器可达0.81；未加定位的编码器在双人场景中仅0.05。二者结合后达到正确区域的基准（0.96），并泛化至多说话人音视频。核心识别模块无需训练，可在任意冻结模型上复现编码器性能，注册成本为O(1)。在PerceptMem数据集（12个领域，1,080项任务）上，感知身份受容量限制，精确事实受绑定限制：身份应存于参数化库，事实存于文本库。两者可无缝组合：兼具两种记忆的智能体不仅能记住用户说了什么，还能记住他们是谁。
原文摘要 · Abstract (English)
A personalized agent needs a user memory: a persistent model of who its user is. Today it is almost always text -- transcripts and captions retrieved by similarity. This serves the captionable half of a person ("my cat is named Bibi"), but discards the perceptual half no caption can hold: how a voice sounds, how a face reads across age and lighting, how tired someone sounds. We measure this loss across five modalities: a strong caption-based re-identifier recovers as little as 0.11 of a dedicated encoder's recall, collapsing toward chance on non-nameable signals. We instead ground perceptual memory in the model, decomposing recall into two subproblems: a vision-language model grounds the referent in context (what and where), and a dedicated encoder extracts an identity key (who), stored as one inline token read by attention at generation with no external round-trip. Neither suffices alone -- the VLM identifies cross-age faces at only 0.54 recall where a face encoder reaches 0.81, and an ungrounded encoder recognizes a two-person-scene referent at 0.05 -- yet together they reach correct-region oracle (0.96), generalizing to multi-speaker audio and video. The recognition core is training-free: it reproduces the encoder's recall on any frozen model at O(1) registration cost. On PerceptMem (12 domains, 1,080 tasks) perceptual identity is capacity-limited while exact facts are binding-limited: identity belongs in a parametric bank, facts in a text store. The two memories compose cleanly: an agent with both can remember not only what its user said, but also what they are like.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。