测试大模型在干扰下整合不同经验做决策的能力，发现超长上下文会显著降低表现。
Probing the Capacity of Language Model Agents to Operationalize Disparate Experiential Context Despite Distraction
- 构建带历史记录和干扰项的OEDD评测集，模拟复杂决策场景。
- 当上下文超过1615词且存在误导信息时，模型表现低于随机水平。
- 适用于评估大模型在真实场景中处理复杂经验的能力。
大型语言模型代理在多个领域展现出潜力，常需基于输入提示中的累积经验进行推理。为此，我们提出OEDD（Operationalize Experience Despite Distraction）语料库，包含经人工标注验证的场景与预设代理历史，要求代理在存在干扰项的情况下，基于分散的经验信息做出决策。使用最小链式思维提示策略，评估GPT-3.5 Turbo、GPT-4o和Gemini 1.5 Pro三款先进模型，结果表明：当输入上下文包含超过1615个词的历史交互、关键决策依据需从两个不同环境前提中推导出合理结论、且存在一个琐碎但具有干扰性的虚假事实时，所有模型在选择更优行动方案上的表现均低于随机猜测。代码与测试语料库已公开于https://github.com/sonnygeorge/OEDD。
原文摘要 · Abstract (English)
Large language model (LLM) agents show promise in an increasing number of domains. In many proposed applications, it is expected that the agent reasons over accumulated experience presented in an input prompt. We propose the OEDD (Operationalize Experience Despite Distraction) corpus, a human-annotator-validated body of scenarios with pre-scripted agent histories where the agent must make a decision based on disparate experiential information in the presence of a distractor. We evaluate three state-of-the-art LLMs (GPT-3.5 Turbo, GPT-4o, and Gemini 1.5 Pro) using a minimal chain-of-thought prompting strategy and observe that when (1) the input context contains over 1,615 tokens of historical interactions, (2) a crucially decision-informing premise is the rightful conclusion over two disparate environment premises, and (3) a trivial, but distracting red herring fact follows, all LLMs perform worse than random choice at selecting the better of two actions. Our code and test corpus are publicly available at: https://github.com/sonnygeorge/OEDD .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。