通过多轮强化学习揭示大模型智能体的自我演化机制。
RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning
- 提出星型策略优化框架，实现轨迹级智能体强化学习
- 发现奖励方差突增导致循环陷阱，提出稳定化训练方案
- 强调细粒度推理奖励对生成真实思考至关重要
将大语言模型训练为交互式智能体面临长周期决策与随机环境反馈等挑战。尽管强化学习在静态任务中取得进展，多轮智能体强化学习仍研究不足。本文提出星型策略优化（StarPO）——一种面向轨迹级智能体强化学习的通用框架，并构建模块化系统RAGEN用于训练与评估。在四个风格化环境中研究发现：第一，传统训练存在奖励方差突增与梯度激增的“回声陷阱”现象，我们通过引入轨迹过滤、批判器融合与梯度稳定化改进的StarPO-S解决；第二，强化学习采样需多样化初始状态、适中交互粒度与更高频率采样；第三，若缺乏细粒度、推理感知的奖励信号，智能体难以产生真实推理，可能表现出浅层策略或幻觉思维。代码与环境已开源。
原文摘要 · Abstract (English)
Training large language models (LLMs) as interactive agents presents unique challenges including long-horizon decision making and interacting with stochastic environment feedback. While reinforcement learning (RL) has enabled progress in static tasks, multi-turn agent RL training remains underexplored. We propose StarPO (State-Thinking-Actions-Reward Policy Optimization), a general framework for trajectory-level agent RL, and introduce RAGEN, a modular system for training and evaluating LLM agents. Our study on four stylized environments reveals three core findings. First, our agent RL training shows a recurring mode of Echo Trap where reward variance cliffs and gradient spikes; we address this with StarPO-S, a stabilized variant with trajectory filtering, critic incorporation, and gradient stabilization. Second, we find the shaping of RL rollouts would benefit from diverse initial states, medium interaction granularity and more frequent sampling. Third, we show that without fine-grained, reasoning-aware reward signals, agent reasoning hardly emerge through multi-turn RL and they may show shallow strategies or hallucinated thoughts. Code and environments are available at https://github.com/RAGEN-AI/RAGEN.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。