评测从回忆文本中推断隐含实体的难题,推动记忆理解技术发展。
IRC-Bench: Recognizing Entities from Contextual Cues in First-Person Reminiscences
- 构建跨片段上下文线索的实体推理基准,模拟真实回忆表达方式
- 在开放世界下模型最佳准确率达38.94%,闭合世界检索达71.49%命中率
- 适合研究记忆智能、对话系统与知识推理的开发者使用
人们在讲述个人记忆时,常通过上下文线索间接提及人、地、事,而非直接命名。这类隐含指代是回忆叙事的核心,在治疗、档案和社交场景中广泛应用。其计算挑战在于需从分散的非连续句段中推断目标实体,而非局部提及。我们提出IRC-Bench,即隐含回忆上下文基准,用于评估回忆转录中的隐含实体识别。该基准针对非局部性问题:实体线索分布于多个不连续语句中,不同于命名实体识别、实体链接或共指消解。IRC-Bench包含25,136个样本,源自1,994份跨11个主题领域的转录文本,覆盖12,337个与Wikidata关联的实体。每个样本配对一个显式提及目标实体的“实体锚定叙事”与一个移除直接提及的“实体省略叙事”。我们评估了19种配置,涵盖LLM生成、密集检索、RAG与微调。在开放世界设置中,QLoRA适配的Llama 3.1 8B表现最佳(精确匹配38.94%,Jaccard得分51.59%);而微调的DPR在闭合世界检索中领先(Hit@1为35.38%,Hit@10为71.49%)。数据、代码与评估工具已开源。
原文摘要 · Abstract (English)
When people recount personal memories, they often refer to people, places, and events indirectly, relying on con-textual cues rather than explicit names. Such implicit references are central to reminiscence narratives: first-person accounts of lived experience used in therapeutic, archival, and social settings. They pose a difficult computational problem because the intended entity must be inferred from dispersed narrative evidence rather than from a local mention. We introduce IRC-Bench, the Implicit Reminiscence Context Benchmark, for evaluating implicit entity recognition in reminiscence transcripts. The benchmark targets non-locality: entity-identifying cues are distributed across multiple, non-contiguous clauses, unlike named entity recognition, entity linking, or coreference resolution. IRC-Bench comprises 25,136 samples constructed from 12,337 Wikidata-linked entities across 1,994 transcripts spanning 11 thematic domains. Each sample pairs an Entity-Grounded Narrative, in which the target entity is ex-plicitly mentioned, with an Entity-Elided Narrative, in which direct mentions are removed. We evaluate 19 config-urations across LLM generation, dense retrieval, RAG, and fine-tuning. QLoRA-adapted Llama 3.1 8B performs best in the open-world setting (38.94% exact match; 51.59% Jaccard), while fine-tuned DPR leads closed-world retrieval (35.38% Hit@1; 71.49% Hit@10). We release IRC-Bench with data, code, and evaluation tools.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。