CoDA用分层设计解决大模型推理时的上下文爆炸问题。
CoDA: A Context-Decoupled Hierarchical Agent with Reinforcement Learning
- 将规划与执行分离,分属不同上下文环境,避免信息过载。
- 在多跳问答任务中超越现有基线,长上下文下性能稳定不下降。
- 适合需要长期推理和复杂任务分解的应用场景。
基于强化学习的大语言模型代理在解决复杂多步骤任务方面展现出巨大潜力,但常因“上下文爆炸”而失效——长期文本输出堆积导致模型上下文窗口超限,引发推理失败。为此,我们提出CoDA,一种上下文解耦的分层代理框架。该框架采用单一共享的LLM主干,通过学习两种独立、上下文隔离的角色:高层规划器在精炼的战略上下文中分解任务;底层执行器在临时、隔离的工作空间中处理工具交互。我们使用PECO(规划器-执行器协同优化)方法端到端训练该统一代理,以轨迹级奖励联合优化两个角色,通过上下文相关的策略更新促进无缝协作。大量实验表明,CoDA在复杂多跳问答基准测试中显著优于现有先进基线,在长上下文场景中表现出强鲁棒性,性能稳定,而其他基线则严重退化,进一步验证了其分层设计在缓解上下文过载方面的有效性。
原文摘要 · Abstract (English)
Large Language Model (LLM) agents trained with reinforcement learning (RL) show great promise for solving complex, multi-step tasks. However, their performance is often crippled by "Context Explosion", where the accumulation of long text outputs overwhelms the model's context window and leads to reasoning failures. To address this, we introduce CoDA, a Context-Decoupled hierarchical Agent, a simple but effective reinforcement learning framework that decouples high-level planning from low-level execution. It employs a single, shared LLM backbone that learns to operate in two distinct, contextually isolated roles: a high-level Planner that decomposes tasks within a concise strategic context, and a low-level Executor that handles tool interactions in an ephemeral, isolated workspace. We train this unified agent end-to-end using PECO (Planner-Executor Co-Optimization), a reinforcement learning methodology that applies a trajectory-level reward to jointly optimize both roles, fostering seamless collaboration through context-dependent policy updates. Extensive experiments demonstrate that CoDA achieves significant performance improvements over state-of-the-art baselines on complex multi-hop question-answering benchmarks, and it exhibits strong robustness in long-context scenarios, maintaining stable performance while all other baselines suffer severe degradation, thus further validating the effectiveness of our hierarchical design in mitigating context overload.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。