arXiv:2510.11184cs.LGcs.CL2025-10被引 5

让AI在解题中不断规划-执行-反思,跨领域通用性更强。

Reinforcement Learning for Tool-Integrated Interleaved Thinking towards Cross-Domain Generalization

  • 用持续循环的计划-执行-反思机制,让模型根据工具输出自我修正。
  • 仅在数学任务训练,却在多个领域达到顶尖表现,效率高。
  • 适合需要长期推理与跨域迁移的复杂任务研究者。

大语言模型在推理和工具使用方面展现出显著能力,但工具增强型强化学习在跨域泛化方面仍面临挑战。传统方法将工具使用视为线性或孤立事件,在从限定领域(如数学)迁移到开放任务时容易失效。本文研究仅在数学问题求解上训练的LLM智能体的跨域泛化能力。为此提出R{\textbf{I}}TE(Reinforcement Learning for Interleaved Tool Execution),通过连续的“计划-行动-反思”循环,使模型能基于中间工具输出进行推理并自我修正,提升长程任务鲁棒性。为有效训练该复杂交错策略,引入Dr. GRPO优化目标,采用分词级损失聚合与重要性采样缓解奖励稀疏与信用分配方差问题。同时采用双组件奖励系统与在线回放过滤动态课程,保障结构完整性和样本效率。大量实验表明,该方法虽仅在数学任务上训练,却在多种推理领域实现最优性能,展现高令牌效率与强泛化能力。

原文摘要 · Abstract (English)

Recent advances in large language models (LLMs) have demonstrated remarkable capabilities in reasoning and tool utilization. However, the generalization of tool-augmented reinforcement learning (RL) across diverse domains remains a significant challenge. Standard paradigms often treat tool usage as a linear or isolated event, which becomes brittle when transferring skills from restricted domains (e.g., mathematics) to open-ended tasks. In this work, we investigate the cross-domain generalization of an LLM agent trained exclusively on mathematical problem-solving. To facilitate robust skill transfer, we propose a {\textbf{R}einforcement Learning for \textbf{I}nterleaved \textbf{T}ool \textbf{E}xecution (RITE)}. Unlike traditional methods, RITE enforces a continuous ``Plan-Action-Reflection'' cycle, allowing the model to ground its reasoning in intermediate tool outputs and self-correct during long-horizon tasks. To effectively train this complex interleaved policy, we introduce {Dr. GRPO}, a robust optimization objective that utilizes token-level loss aggregation with importance sampling to mitigate reward sparsity and high-variance credit assignment. Furthermore, we employ a dual-component reward system and dynamic curriculum via online rollout filtering to ensure structural integrity and sample efficiency. Extensive experiments reveal that our approach, despite being trained solely on math tasks, achieves state-of-the-art performance across diverse reasoning domains, demonstrating high token efficiency and strong generalization capabilities.

强化学习工具使用跨域泛化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。