简单遮蔽观察记录比大模型摘要更省成本且效果相当
The Complexity Trap: Simple Observation Masking Is as Efficient as LLM Summarization for Agent Context Management
- 用遮蔽旧观测代替复杂摘要，降低上下文开销
- 成本减半，任务解决率与摘要法持平甚至略优
- 适合追求高效推理的智能体开发与部署
基于大语言模型（LLM）的智能体通过迭代推理、探索和工具使用完成复杂任务，但会产生冗长昂贵的上下文历史。当前主流软件工程智能体如OpenHands或Cursor采用LLM摘要来缓解此问题，但其复杂性是否带来实际性能提升尚不明确。我们在SWE-agent上对SWE-bench Verified数据集进行了五种不同模型配置的系统对比，并初步验证了结果在OpenHands框架下的可泛化性。结果表明，仅通过简单遮蔽旧环境观测的方法，可使成本减半，同时保持甚至略微超过LLM摘要的任务解决率。此外，我们提出一种新型混合策略，相较单纯遮蔽或LLM摘要分别进一步降低7%和11%的成本。研究质疑了纯依赖LLM摘要的趋势，并揭示了提升效率-效果边界的潜在空间。代码与数据已公开以支持复现。
原文摘要 · Abstract (English)
Large Language Model (LLM)-based agents solve complex tasks through iterative reasoning, exploration, and tool-use, a process that can result in long, expensive context histories. While state-of-the-art Software Engineering (SE) agents like OpenHands or Cursor use LLM-based summarization to tackle this issue, it is unclear whether the increased complexity offers tangible performance benefits compared to simply omitting older observations. We present a systematic comparison of these approaches within SWE-agent on SWE-bench Verified across five diverse model configurations. Moreover, we show initial evidence of our findings generalizing to the OpenHands agent scaffold. We find that a simple environment observation masking strategy halves cost relative to the raw agent while matching, and sometimes slightly exceeding, the solve rate of LLM summarization. Additionally, we introduce a novel hybrid approach that further reduces costs by 7% and 11% compared to just observation masking or LLM summarization, respectively. Our findings raise concerns regarding the trend towards pure LLM summarization and provide initial evidence of untapped cost reductions by pushing the efficiency-effectiveness frontier. We release code and data for reproducibility.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。