arXiv:2604.24893cs.CV2026-04中稿 · CVPR

让记忆系统通过用户反馈迭代优化,更准定位视频中的物品位置。

Interactive Episodic Memory with User Feedback

论文配图:Interactive Episodic Memory with User Feedback
图 1 · 摘自论文原文
  • 引入交互式问答反馈机制,支持用户修正或补充查询信息
  • 在三个基准上超越现有模型,且效率优于商用大模型
  • 轻量训练+模块化设计,可无缝集成到现有记忆系统中

在自然语言查询的场景记忆(EM-NLQ)中,用户需从长段第一人称视角视频中检索特定事件(如“杯子放哪了?”)。但查询常模糊或不完整,导致错误回答。现有方法多为单次推理,难以应对真实场景。本文提出带问题与反馈的场景记忆任务(EM-QnF),允许用户对初始结果提供反馈或补充信息(如“不是白色的,是蓝色的大杯子”),从而引导模型迭代优化。为此,我们构建了基于反馈的交互数据集,并提出轻量级训练方案,避免复杂序列优化。同时设计即插即用的反馈对齐模块(FALM),使已有模型可有效融入用户反馈。实验显示,该方法在三个挑战性基准上显著优于当前最优模型,性能媲美商业大视觉语言模型,且计算高效。真人反馈评估表明其具备良好的实际泛化能力。

原文摘要 · Abstract (English)

In episodic memory with natural language queries (EM-NLQ), a user may ask a question (e.g., "Where did I place the mug?") that requires searching a long egocentric video, captured from the user's perspective, to find the moment that answers it. However, queries can be ambiguous or incomplete, leading to incorrect responses. Current methods ignore this key aspect and address EM-NLQ in a one-shot setup, limiting their applicability in real-world scenarios. In this work, we address this gap and introduce the Episodic Memory with Questions and Feedback task (EM-QnF). Here, the user can provide feedback on the model's initial prediction or add more information (e.g., "Before this. I'm looking for the big blue mug not the white one"), helping the model refine its predictions interactively. To this end, we collect datasets for feedback-based interaction and propose a lightweight training scheme that avoids expensive sequential optimization. We also introduce a plug-and-play Feedback ALignment Module (FALM) that enables existing EM-NLQ models to incorporate user feedback effectively. Our approach significantly improves over the state of the art on three challenging benchmarks and is better than or competitive with commercial large vision-language models while remaining efficient. Evaluation with human-generated feedback shows that it generalizes well to real-world scenarios.

场景记忆交互式问答用户反馈视觉语言模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。