用故事化提示提升小模型的时间推理能力，效果接近GPT-3.5。
Narrative-of-Thought: Improving Temporal Reasoning of Large Language Models via Recounted Narratives
- 将事件转为代码类，引导小模型生成时间连贯的故事
- 在Schema-11上达最高F1，结构相似性超越GPT-3.5/4
- 无需微调，适合资源有限但需强时序推理的场景
时间推理是人类认知的核心能力，但大语言模型（LLMs）在此任务上仍表现不佳。本文研究时间图生成这一关键任务，发现即使是GPT-3.5/4也面临巨大挑战，而小模型（<10B）性能落后高达50%。针对预算受限场景（如不使用微调），提出新型提示方法Narrative-of-Thought（NoT）：先将事件集转化为Python类，再引导小模型生成具时间锚定的故事，辅助生成时间图。大量实验表明，NoT在多个指标上显著提升，其在Schema-11上的F1达到最高，整体F1与GPT-3.5相当，且在结构相似性上优于GPT-3.5/4。代码已开源。
原文摘要 · Abstract (English)
Reasoning about time and temporal relations is an integral aspect of human cognition, essential for perceiving the world and navigating our experiences. Though large language models (LLMs) have demonstrated impressive performance in many reasoning tasks, temporal reasoning remains challenging due to its intrinsic complexity. In this work, we first study an essential task of temporal reasoning -- temporal graph generation, to unveil LLMs' inherent, global reasoning capabilities. We show that this task presents great challenges even for the most powerful LLMs, such as GPT-3.5/4. We also notice a significant performance gap by small models (<10B) that lag behind LLMs by 50%. Next, we study how to close this gap with a budget constraint, e.g., not using model finetuning. We propose a new prompting technique tailored for temporal reasoning, Narrative-of-Thought (NoT), that first converts the events set to a Python class, then prompts a small model to generate a temporally grounded narrative, guiding the final generation of a temporal graph. Extensive experiments showcase the efficacy of NoT in improving various metrics. Notably, NoT attains the highest F1 on the Schema-11 evaluation set, while securing an overall F1 on par with GPT-3.5. NoT also achieves the best structural similarity across the board, even compared with GPT-3.5/4. Our code is available at https://github.com/launchnlp/NoT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。