提升大模型在跨会话对话中的时间推理能力
TReMu: Towards Neuro-Symbolic Temporal Reasoning for LLM-Agents with Memory in Multi-Session Dialogues
- 用时间感知记忆总结对话事件并生成可检索记忆
- 结合神经符号方法，让模型写代码进行时间计算
- 在多会话场景下将推理准确率从29.83提升至77.67
多会话对话中的时间推理是一个重要但未被充分研究的挑战。为填补这一空白，我们提出了一项新的多会话对话时间推理评估任务，并通过增强LoCoMo数据集对话内容、构建多选题问答对，创建了一个新基准。此外，我们提出了TReMu框架，旨在提升大模型代理在此场景下的时间推理能力。该框架采用时间感知记忆机制，通过时间线摘要生成可检索的记忆，将每轮对话中的事件及其推断日期进行总结。同时，集成神经符号时间推理，使大模型生成Python代码执行时间计算并选择答案。在主流大模型上的实验表明，该基准具有挑战性，所提框架显著优于基线方法：GPT-4o在标准提示下的准确率为29.83，而采用本方法后提升至77.67，验证了其在多会话时间推理中的有效性。
原文摘要 · Abstract (English)
Temporal reasoning in multi-session dialogues presents a significant challenge which has been under-studied in previous temporal reasoning benchmarks. To bridge this gap, we propose a new evaluation task for temporal reasoning in multi-session dialogues and introduce an approach to construct a new benchmark by augmenting dialogues from LoCoMo and creating multi-choice QAs. Furthermore, we present TReMu, a new framework aimed at enhancing the temporal reasoning capabilities of LLM-agents in this context. Specifically, the framework employs time-aware memorization through timeline summarization, generating retrievable memory by summarizing events in each dialogue session with their inferred dates. Additionally, we integrate neuro-symbolic temporal reasoning, where LLMs generate Python code to perform temporal calculations and select answers. Experimental evaluations on popular LLMs demonstrate that our benchmark is challenging, and the proposed framework significantly improves temporal reasoning performance compared to baseline methods, raising from 29.83 on GPT-4o via standard prompting to 77.67 via our approach and highlighting its effectiveness in addressing temporal reasoning in multi-session dialogues.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。