压缩上下文会隐藏增加交互成本,任务完成率却看不出变化。
What Does Context Compression Cost an Agent? Interaction Costs Unrevealed by Task-Completion Metrics
- 通过控制实验测量状态丢失后重获取的交互开销。
- 压缩导致检索调用增加近3倍,但任务完成率无显著变化。
- 适合关注模型运行效率与真实交互成本的研究者。
任务完成率是评估上下文压缩的标准指标,但存在缺陷:压缩可能因迫使智能体重新获取丢失的状态而增加交互成本,同时任务完成率统计上保持不变。本文设计了一种受控的运行时测量协议,在固定24轮回合的确定性规划环境中,测试工具使用型智能体的重获取成本。通过改变压缩强度,对比丢弃型与保留事实型操作符,利用受控的虚拟干预恢复丢失状态,并将工具调用分解为检索与执行两部分。在两个任务场景中评估三种模型。所有六组对比中检索调用均上升,且几乎全部由新增交互贡献;五组在霍尔姆校正后仍显著。在预设5倍压缩点,所有组别任务完成率均不显著变化。DeepSeek仅在10倍压缩时出现显著下降;GPT-5.5表现最清晰:任务完成率从80%升至85%(p=1.0),检索调用从21.0增至63.9次(p=.002)。状态保留干预进一步区分了状态数量、类型与内容有效性:随机选择与离线事后虚拟机相当,而用语义无关内容替换保留的D状态使检索增加57%(p<.001),但完成率无显著变化。在第二个环境ALFWorld中,滑动压缩未引发检索激增,表明该重获取特征依赖于环境而非压缩本身。总体而言,当执行相关状态缺失需重获取时,压缩会引入隐藏的交互成本,而仅依赖任务完成率可能无法揭示此类代价。
原文摘要 · Abstract (English)
Task completion is the standard metric for evaluating context compression, yet it is incomplete: compression can increase an agent's interaction cost by forcing it to reacquire dropped state while leaving completion statistically unchanged. We introduce a controlled runtime measurement protocol for reacquisition cost in a bounded-horizon tool-using agent. The agent acts in a deterministic planning environment under a fixed 24-turn horizon. We vary compression severity, compare a dropping operator with a fact-preserving operator, restore dropped state through controlled oracle interventions, and decompose tool calls into retrieval and execution. We evaluate three models across two task regimes. Retrieval calls increase in all six model-regime comparisons and account for almost all added interaction; five of six remain significant after Holm correction. At the prespecified 5x comparison point, completion changes are not significant in any cell. DeepSeek shows a significant completion drop only at 10x compression. GPT-5.5 is the clearest case: completion changes from 80% to 85% (p = 1.0) while retrieval increases from 21.0 to 63.9 calls (p = .002). Retention interventions further separate state quantity, state type, and content validity. Random selection is comparable to an offline hindsight oracle, while replacing retained D-state with semantically irrelevant content increases retrieval by 57% (p < .001) without a significant completion change. In a second environment, ALFWorld, sliding compression produces no retrieval surge, showing that the reacquisition signature is environment-dependent rather than intrinsic to shortening context. Overall, compression can impose hidden interaction costs when execution-relevant state becomes absent and must be reacquired, while completion alone may not expose those costs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。