让强化学习系统自动优化环境、策略和奖励模型，提升大模型任务表现。
RLAnything: Forge Environment, Policy, and Reward Model in Completely Dynamic RL System
- 三模型闭环动态优化：环境、策略、奖励模型相互增强。
- 在多个任务上显著提升性能，最高增益达18.7%。
- 无需人工标注，优化后的奖励信号优于人类标签结果。
我们提出RLAnything，一种通过闭环优化动态构建环境、策略和奖励模型的强化学习框架，可放大学习信号并增强任意大语言模型或智能体场景下的整体性能。策略通过分步与最终结果反馈联合训练，奖励模型则通过一致性反馈协同优化，进而反哺策略训练。基于理论驱动的自动环境适应机制，利用两者批评反馈实现经验学习，进一步提升奖励与策略模型训练效果。实验表明，每个新增组件均持续提升系统表现，RLAnything在多个代表性大模型与智能体任务中取得显著进步：在OSWorld上使Qwen3-VL-8B-Thinking提升9.1%，在AlfWorld和LiveBench上分别使Qwen2.5-7B-Instruct提升18.7%和11.9%。此外，优化后的奖励信号优于依赖人工标注的结果。代码已开源。
原文摘要 · Abstract (English)
We propose RLAnything, a reinforcement learning framework that dynamically forges environment, policy, and reward models through closed-loop optimization, amplifying learning signals and strengthening the overall RL system for any LLM or agentic scenarios. Specifically, the policy is trained with integrated feedback from step-wise and outcome signals, while the reward model is jointly optimized via consistency feedback, which in turn further improves policy training. Moreover, our theory-motivated automatic environment adaptation improves training for both the reward and policy models by leveraging critic feedback from each, enabling learning from experience. Empirically, each added component consistently improves the overall system, and RLAnything yields substantial gains across various representative LLM and agentic tasks, boosting Qwen3-VL-8B-Thinking by 9.1% on OSWorld and Qwen2.5-7B-Instruct by 18.7% and 11.9% on AlfWorld and LiveBench, respectively. We also that optimized reward-model signals outperform outcomes that rely on human labels. Code: https://github.com/Gen-Verse/Open-AgentRL
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。