LLM处理图任务时，信息距离越远表现越差，影响推理能力。
Lost-in-Distance: Impact of Contextual Proximity on LLM Performance in Graph Tasks
- 通过分析节点间连接位置，揭示上下文距离影响模型性能。
- 信息距离增加导致准确率最高下降6倍，与模型大小无关。
- 发现'丢失在距离中'与'丢失在中间'是独立现象，适合图神经网络研究者。
尽管大语言模型（LLMs）取得了显著进展，但在处理复杂图任务时仍存在盲区，难以有效检索和处理相关上下文数据。本文证明，在超出‘针尖对草堆’场景的图任务中——解决此类问题需跨多个子问题联合推理和交叉引用——模型表现受相关信息在上下文中的相对位置影响，这种现象称为‘丢失在距离中’（lost-in-distance）。我们考察了两个基础图任务：识别两节点间的公共连接，以及评估三节点间的相似性，并发现模型表现显著依赖于公共边的相对位置。使用三种公开可用的LLMs，结合不同图编码技术（graph encoding techniques）进行评估。提出‘丢失在距离中’现象的形式化表达，证明其与‘丢失在中间’现象独立存在。结果显示，随着节点连接间距离增加，模型准确率最高下降6倍，且该现象不受图编码方式或模型规模影响。
原文摘要 · Abstract (English)
Despite significant advancements, Large Language Models (LLMs) exhibit blind spots that impair their ability to retrieve and process relevant contextual data effectively. We demonstrate that LLM performance in graph tasks with complexities beyond the "needle-in-a-haystack" scenario-where solving the problem requires cross-referencing and reasoning across multiple subproblems jointly-is influenced by the proximity of relevant information within the context, a phenomenon we term "lost-in-distance". We examine two fundamental graph tasks: identifying common connections between two nodes and assessing similarity among three nodes, and show that the model's performance in these tasks significantly depends on the relative positioning of common edges. We evaluate three publicly available LLMs using various graph encoding techniques that represent graph structures for LLM input. We propose a formulation for the lost-in-distance phenomenon and demonstrate that lost-in-distance and lost-in-the middle phenomenas occur independently. Results indicate that model accuracy can decline by up to 6x as the distance between node connections increases, independent of graph encoding and model size.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。