解决大模型长时序任务中上下文折叠的不稳定性问题
FoldAct: Efficient and Stable Context Folding for Long-Horizon Search Agents
- 分离摘要与动作的损失计算，避免梯度稀释
- 通过完整上下文一致性损失减少分布漂移
- 选择性分段训练提升效率，提速5.19倍
大型语言模型在长时序强化学习中面临上下文无限增长带来的可扩展性挑战，现有上下文折叠方法将摘要动作视为标准动作，忽视了摘要会改变智能体未来的观测空间，导致策略依赖的非平稳观测分布，违背强化学习基本假设。这引发三大问题：（1）梯度稀释，摘要令牌获得不足训练信号；（2）自条件依赖，策略更新改变摘要分布，形成训练崩溃的恶性循环；（3）每轮处理唯一上下文导致计算成本高。我们提出FoldAct，通过三项创新解决：分离损失计算以获得独立梯度信号，全上下文一致性损失降低分布偏移，选择性分段训练降低计算开销。该方法使长时序搜索智能体在上下文折叠下实现稳定训练，同时提升效率，达到5.19倍加速。
原文摘要 · Abstract (English)
Long-horizon reinforcement learning (RL) for large language models faces critical scalability challenges from unbounded context growth, leading to context folding methods that compress interaction history during task execution. However, existing approaches treat summary actions as standard actions, overlooking that summaries fundamentally modify the agent's future observation space, creating a policy-dependent, non-stationary observation distribution that violates core RL assumptions. This introduces three fundamental challenges: (1) gradient dilution where summary tokens receive insufficient training signal, (2) self-conditioning where policy updates change summary distributions, creating a vicious cycle of training collapse, and (3) computational cost from processing unique contexts at each turn. We introduce \textbf{FoldAct}\footnote{https://github.com/SHAO-Jiaqi757/FoldAct}, a framework that explicitly addresses these challenges through three key innovations: separated loss computation for independent gradient signals on summary and action tokens, full context consistency loss to reduce distribution shift, and selective segment training to reduce computational cost. Our method enables stable training of long-horizon search agents with context folding, addressing the non-stationary observation problem while improving training efficiency with 5.19$\times$ speedup.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。