让大模型在长任务中自动压缩上下文,提升执行效率。
CompactionRL: Reinforcement Learning with Context Compaction for Long-Horizon Agents
- 用强化学习联合优化任务执行与上下文压缩
- 在多个编码任务上实现最高7.0点的性能提升
- 适合需要长时间推理的大模型研究者使用
长时序智能体大模型受限于有限的上下文窗口,长期交互轨迹常超出最大上下文长度。上下文压缩通过总结历史状态,使模型能在压缩后继续执行,但将其融入强化学习仍不充分。本文提出 CompactionRL,一种支持上下文压缩的强化学习策略,通过分词级损失归一化与跨轨迹广义优势估计,联合优化任务执行与摘要生成。该方法使大模型能从压缩后的长时序轨迹中学习。我们在开源模型上训练并验证,结果表明:基于 GLM-4.5-Air(106B-A30B)的 CompactionRL 在 SWE-bench Verified 上达到 66.8% Pass@1,较原模型提升 7.0 点;在 Terminal-Bench 2.0 上达 24.5%,提升 3.1 点。基于 GLM-4.7-Flash(30B-A3B)的版本分别提升 5.5 和 6.8 点,分别达 56.0% 和 20.2%。CompactionRL 已部署于开放模型 GLM-5.2(750B-A40B)的强化学习训练流程中。
原文摘要 · Abstract (English)
Long-horizon agentic LLMs are increasingly limited by finite context windows, as extended interaction trajectories can exceed the maximum context length before a task is completed. Context compaction offers a natural solution by summarizing previous interaction states and continuing the rollout under a compressed context, but incorporating compaction into reinforcement learning remains underexplored. We propose CompactionRL, a reinforcement learning strategy to train long-horizon agentic LLMs with context compaction. Our approach jointly optimizes task execution and summary generation with token-level loss normalization and cross-trajectory generalized advantage estimation. This design enables the LLM agents to learn from compacted long-horizon trajectories. We train CompactionRL on top of open models and observe consistent performance gains on agentic coding tasks. CompactionRL enables the open GLM-4.5-Air model (106B-A30B) to achieve Pass@1 scores of 66.8% on SWE-bench Verified and 24.5% on Terminal-Bench 2.0, with absolute gains of 7.0 and 3.1 points, respectively. Built upon GLM-4.7-Flash (30B-A3B), CompactionRL improves Pass@1 by 5.5 and 6.8 points, reaching 56.0% on SWE-bench Verified and 20.2% on Terminal-Bench 2.0, respectively. CompactionRL is thus deployed in the RL pipeline for training the open GLM-5.2 model (750B-A40B).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。