提升第一人称记忆系统搜索精度与时间感知能力
EgoCITE: Context-Augmented Indexing and Time-Aware Retrieval for Long-Horizon Egocentric Memory
- 用多模态上下文生成自包含的原子记忆索引
- 结合语义与时间相关性，提升检索准确率4.4%~14.2%
- 适合长期第一人称记忆与智能体问答场景
长时序第一人称记忆将连续的第一视角视频和音频转化为可检索的过往经历记录。我们指出现有系统的两大瓶颈：基于上下文贫乏描述的索引在智能体搜索中不可靠，且检索过程忽略问题的时间意图。为此，我们提出EgoCITE（第一人称上下文增强索引与时间感知证据检索），一个面向第一人称问答的长时序智能体记忆框架。EgoCITE包含三个组件：EgoScheme利用局部多模态上下文，将零散的视频描述和语音转录文本转换为自包含的原子记忆索引；EgoIndex将动作、活动、话语和对话表征整合为多视图、多粒度的可搜索记忆索引；EgoRetrv结合语义搜索与条件化时间相关性评分，实现证据筛选。我们在EgoLifeQA、EgoMem和EgoR1-Bench上评估，相比基线智能体记忆系统，EgoCITE准确率提升至少4.4%~14.2%，同时成本降低36×。
原文摘要 · Abstract (English)
Long-horizon egocentric memory transforms continuous first-person video and audio into a searchable record of past experiences. We demonstrate two bottlenecks in existing systems: indices built from context-poor captions are unreliable for agentic search, while retrieval ignores a question's temporal intent. To address both bottlenecks, we introduce EgoCITE (Egocentric Context-augmented Indexing and Time-aware Evidence retrieval), a long-horizon agentic memory framework for egocentric QA. EgoCITE comprises three components. EgoScheme uses local multimodal context to turn fragmentary video captions and speech transcripts into self-contained atomic memory indices. EgoIndex organizes complementary action, activity, utterance, and conversation representations into searchable multi-view memory indices at multiple granularities. EgoRetrv combines semantic search with question-conditioned temporal relevance scoring and curation of retrieved evidence. We evaluate EgoCITE on EgoLifeQA, EgoMem, and EgoR1-Bench in terms of answer accuracy and target-event retrieval alignment. EgoCITE improves accuracy over agentic memory baselines by at least 4.4--14.2% while achieving 36$\times$ lower cost than long-context LLM agents.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。