首次为大模型上下文压缩建立理论框架,揭示生成优于选择的原理。
Context Compaction Theory

- 提出上下文选择与生成两类压缩游戏,建模实际算法策略。
- 证明生成压缩预算等价于单向通信复杂度,可直接复用已有理论边界。
- 实证显示生成法比选择法更高效,适用于优化智能体状态管理。
大语言模型(LLMs)具有固定的上下文窗口,即单次推理所能处理的最大输入长度。智能体在调用大模型时,依赖上下文压缩技术将自身状态压缩至窗口内。尽管该技术广泛应用,却缺乏正式分析。本文首次系统研究上下文压缩,提出两个博弈框架:上下文选择博弈模拟从累积状态中筛选子集的算法;上下文生成博弈模拟通过有限长度消息摘要状态的算法。我们证明上下文生成博弈等价于单向通信复杂度。在给定误差下,回答一组查询所需的最小压缩预算,等于对应通信问题的单向通信复杂度。已知的通信复杂度界可直接应用于上下文压缩。此外,上下文选择博弈对应一类受限的单向通信协议。若生成法优于选择法,则意味着两类通信协议间存在差距。我们证明存在查询集合使得生成法所需预算严格小于选择法。该等价关系还可用于评估部署算法相对于最优策略的性能。以 Anthropic 的上下文压缩接口为例,我们对其在集合成员查询上的表现进行了案例研究。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have a bounded context window. The context window is the maximum input size an LLM can consume for a single inference. AI agents rely on a process called context compaction to fit their state within the context window when calling an LLM. Despite its ubiquity, context compaction has received essentially no formal analysis. In this paper, we initiate a formal study of context compaction. We first introduce a framework consisting of two games that capture the two algorithmic strategies for context compaction used by contemporary AI agents in practice. The Context Selection Game models context compaction algorithms that select a subset of an agent's accumulated state to retain. The Context Generation Game models context compaction algorithms that summarize an agent's state by an arbitrary message of bounded length. We then prove an equivalence between the Context Generation Game and one-way communication complexity. The minimum context compaction budget for answering a set of queries within a target error is equal to the one-way communication complexity of the induced communication problem at the same error. Known bounds from communication complexity therefore transfer directly to context compaction. We also show that the Context Selection Game corresponds to a restricted class of one-way communication protocols. Any gap between selection and generation is therefore a gap between two classes of communication protocols. We prove that there exists a set of queries for which generation needs strictly less budget than selection. The equivalence between the Context Generation Game and one-way communication also lets us measure how well a deployed context compaction algorithm performs on a query relative to the optimal strategy. As an example, we present a case study that evaluates Anthropic's context compaction endpoint on set membership queries.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。