大模型推理失败源于内部信息传输带宽不足，该研究提出新框架解释并验证了这一现象。
Lost in Transmission: When and Why LLMs Fail to Reason Globally
- 构建BAPO模型模拟注意力头的信息传输瓶颈，揭示推理障碍根源
- 实验显示GPT-4o等在小型复杂任务上已出现失败，印证理论预测
- 链式思维可突破传输瓶颈，为改进推理提供新路径
尽管取得诸多成功，基于Transformer的大语言模型在需要对长输入进行复杂推理的任务上仍表现不佳。我们认为这些失败源于模型内部信息传递的容量限制。为此，提出有界注意力前缀预言机（BAPO）模型，用于建模注意力头这一内部通信机制的带宽约束。理论分析表明，图可达性等重要推理问题需高通信带宽才能解决，这类问题称为BAPO-hard。实验验证了理论预测：GPT-4o、Claude和Gemini在BAPO-easy任务中表现良好，但在相对小规模的BAPO-hard任务上已失败。此外，我们证明链式思维（CoT）能将任意BAPO-hard问题转化为BAPO-easy问题。研究结果为大模型的关键失败提供了系统性解释，并指明了缓解带宽限制的架构与推理方法方向。
原文摘要 · Abstract (English)
Despite their many successes, transformer-based large language models (LLMs) continue to struggle with tasks that require complex reasoning over large parts of their input. We argue that these failures arise due to capacity limits on the accurate flow of information within LLMs. To formalize this issue, we introduce the bounded attention prefix oracle (BAPO) model, a new computational framework that models bandwidth constraints on attention heads, the mechanism for internal communication in LLMs. We show that several important reasoning problems like graph reachability require high communication bandwidth for BAPOs to solve; we call these problems BAPO-hard. Our experiments corroborate our theoretical predictions: GPT-4o, Claude, and Gemini succeed on BAPO-easy tasks and fail even on relatively small BAPO-hard tasks. BAPOs also reveal another benefit of chain of thought (CoT): we prove that breaking down a task using CoT can turn any BAPO-hard problem into a BAPO-easy one. Our results offer principled explanations for key LLM failures and suggest directions for architectures and inference methods that mitigate bandwidth limits.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。