让AI通过积累解题策略提升数学推理能力,解决重复问题时越用越聪明。
ARISE: Agent Reasoning with Intrinsic Skill Evolution in Hierarchical Reinforcement Learning
- 构建分层强化学习框架,用管理器动态生成并调用解题技能库。
- 在7个数学基准上表现优于现有方法,尤其在陌生题型上提升显著。
- 适合希望提升AI数学推理能力的研究者与开发者参考。
当前语言模型的数学推理主要依赖可验证奖励的强化学习,但现有方法将每个问题视为独立任务,未能利用训练中积累的可复用策略。为此,本文提出ARISE(Agent Reasoning via Intrinsic Skill Evolution),一种分层强化学习框架。该框架包含高层的技能管理者与低层的执行工作者,共享策略。管理者通过结构化总结成功解题路径,动态构建分层技能库,并基于策略选择机制,在后续推理前检索相关技能。分层奖励设计推动推理能力与技能库质量协同进化。在两个基础模型和七个涵盖竞赛数学与Omni-MATH的基准上,ARISE持续优于GRPO系列算法及记忆增强基线,尤其在分布外任务上表现突出。消融实验表明各模块均有效,且技能库质量与推理性能在训练中同步提升。代码已公开于https://github.com/Skylanding/ARISE。
原文摘要 · Abstract (English)
The dominant paradigm for improving mathematical reasoning in language models relies on Reinforcement Learning with verifiable rewards. Yet existing methods treat each problem instance in isolation without leveraging the reusable strategies that emerge and accumulate during training. To this end, we introduce ARISE (Agent Reasoning via Intrinsic Skill Evolution), a hierarchical reinforcement learning framework, in which a shared policy operates both to manage skills at high-level and to generate responses at low-level (denoted as a Skills Manager and a Worker, respectively). The Manager maintains a tiered skill library through a dedicated skill generation rollout that performs structured summarization of successful solution traces (after execution), while employing a policy-driven selection mechanism to retrieve relevant skills to condition future rollouts (before execution). A hierarchical reward design guides the co-evolution of reasoning ability and library quality. Experiments on two base models and seven benchmarks spanning both competition mathematics and Omni-MATH show that ARISE consistently outperforms GRPO-family algorithms and memory-augmented baselines, with particularly notable gains on out-of-distribution tasks. Ablation studies confirm that each component contributes to the observed improvements and that library quality and reasoning performance improve in tandem throughout training. Code is available at \href{https://github.com/Skylanding/ARISE}{https://github.com/Skylanding/ARISE}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。