提出噪声分解框架，揭示长文本处理失败的三大原因
When Does Divide and Conquer Work for Long Context LLM? A Noise Decomposition Framework
- 将长文本错误归为三类噪声：跨块依赖、模型混淆、聚合误差
- 实验验证分块处理在检索、问答等任务中有效，尤其适合大输入
- 解释为何小模型分块处理能超越大模型单次处理
我们研究大语言模型（LLMs）处理长文本的挑战。提出一个理论框架，将长上下文任务的失败模式分为三类：跨块依赖（任务噪声）、随上下文增长的混淆（模型噪声）以及部分结果整合不完善（聚合器噪声）。基于此，分析多智能体分块策略（即把长序列拆成小块分别处理再聚合）何时有效。在检索、问答和摘要等任务上的实验验证了理论分析，并明确了支持分块处理的条件。通过研究模型保真度随输入长度加速下降的现象，解释了为何对大输入，配置分块的小模型反而可超越单次输入的GPT4o。整体上，本文提供了一个系统性的理解框架，强调通过精心设计的分块与聚合策略，可直接实现长上下文的有效处理。
原文摘要 · Abstract (English)
We investigate the challenge of applying Large Language Models (LLMs) to long texts. We propose a theoretical framework that distinguishes the failure modes of long context tasks into three categories: cross-chunk dependence (task noise), confusion that grows with context size (model noise), and the imperfect integration of partial results (aggregator noise). Under this view, we analyze when it is effective to use multi-agent chunking, i.e., dividing a lengthy sequence into smaller chunks and aggregating the processed results of each chunk. Our experiments on tasks such as retrieval, question answering, and summarization confirm both the theoretical analysis and the conditions that favor multi-agent chunking. By exploring the accelerated decay of model fidelity with input length, we also explain why, for large inputs, a weaker model configured with chunk-based processing can surpass a more advanced model like GPT4o applied in a single shot. Overall, we present a principled understanding framework and our results highlight a direct pathway to handling long contexts in LLMs with carefully managed chunking and aggregator strategies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。