通过自我净化轨迹提升小模型的智能体强化学习效果
CLEANER: Self-Purified Trajectories Boost Agentic Reinforcement Learning
- 利用模型自纠正能力在数据收集时直接清除错误路径
- 在AIME24/25等数据集上平均准确率提升6%以上
- 适合资源受限但需高效推理的小规模语言模型
智能体强化学习使大语言模型能使用如Python解释器等工具解决复杂问题。然而，对于参数量受限的模型（如4B–7B），探索阶段常因频繁执行失败产生噪声轨迹，阻碍策略优化。在基于结果的奖励设置下，这种噪声导致严重的信用分配问题，错误动作与成功结果被一同强化。现有方法面临两难：密集奖励易引发奖励劫持，超采样则计算成本过高。为此，我们提出CLEANER。不同于外部过滤方法，CLEANER利用模型内在自纠错能力，在数据收集阶段直接清除含错上下文。核心的相似性感知自适应回滚（SAAR）机制，通过语义相似性自动调节替换粒度，从浅层执行修复到深层推理替换，构建纯净轨迹。训练于这些自净化路径后，模型内化正确推理模式而非错误恢复循环。在AIME24/25、GPQA和LiveCodeBench上的实验显示，相比基线平均准确率提升6%、3%和5%。值得注意的是，仅用三分之一训练步数即达当前最优性能，凸显轨迹净化作为高效智能体强化学习的可扩展方案。代码与模型已开源。
原文摘要 · Abstract (English)
Agentic Reinforcement Learning (RL) has empowered Large Language Models (LLMs) to utilize tools like Python interpreters for complex problem-solving. However, for parameter-constrained models (e.g., 4B--7B), the exploration phase is often plagued by frequent execution failures, creating noisy trajectories that hinder policy optimization. Under standard outcome-based reward settings, this noise leads to a critical credit assignment issue, where erroneous actions are inadvertently reinforced alongside successful outcomes. Existing mitigations face a dilemma: dense rewards often trigger reward hacking, while supersampling incurs prohibitive computational costs. To address these challenges, we propose CLEANER. Distinct from external filtering methods, CLEANER exploits the model's intrinsic self-correction capabilities to eliminate error-contaminated context directly during data collection. At its core, the Similarity-Aware Adaptive Rollback (SAAR) mechanism autonomously constructs clean, purified trajectories by retrospectively replacing failures with successful self-corrections. Based on semantic similarity, SAAR adaptively regulates replacement granularity from shallow execution repairs to deep reasoning substitutions. By training on these self-purified paths, the model internalizes correct reasoning patterns rather than error-recovery loops. Empirical results on AIME24/25, GPQA, and LiveCodeBench show average accuracy gains of 6%, 3%, and 5% over baselines. Notably, CLEANER matches state-of-the-art performance using only one-third of the training steps, highlighting trajectory purification as a scalable solution for efficient agentic RL. Our models and code are available at GitHub
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。