用蒙特卡洛采样实现对黑盒大模型的强化学习优化
Agentic Monte Carlo: Simulating Reinforcement Learning for Black-Box Agents
- 基于贝叶斯推断，直接采样最优行为轨迹
- 在AgentGym上性能超越提示基线和GRPO方法
- 仅需测试时计算资源，适合无法微调的黑盒模型
大语言模型代理分为可微调的开放权重型和仅提供API的黑盒型。后者虽基于顶尖私有大模型，但无法进行参数级优化，导致多数强化学习方法失效。本文提出阿杰尼特蒙特卡洛（AMC），利用强化学习与贝叶斯推断的等价性，直接从黑盒模型的最优策略后验中采样。该后验以固定黑盒模型为先验，通过序贯蒙特卡洛方法结合学习到的价值函数引导代理行为，不修改底层模型。在AgentGym三个不同环境中的实验表明，AMC显著优于提示基线，且随着测试时计算量增加，性能超过组相对策略优化（GRPO）。结果证明了对黑盒大模型进行类强化学习优化的可行性。代码已开源。
原文摘要 · Abstract (English)
LLM agents operate in two distinct regimes: open-weight agents amenable to reinforcement learning (RL) and black-box agents whose behaviour must be controlled purely at test time. Although black-box agents are often backed by state-of-the-art proprietary LLMs, API-only access precludes parameter-level optimization, rendering most RL methods inapplicable. To address this limitation, we turn to a known equivalence between RL and Bayesian inference. We propose Agentic Monte Carlo (AMC) to directly sample from the optimal policy of a black-box agent rather than training it through RL. The optimal policy is a posterior over trajectories whose prior we define as the fixed black-box LLM agent. We employ Sequential Monte Carlo to sample from this posterior by learning a value function to steer the agent while leaving the underlying black-box model unchanged. We validate AMC on three diverse environments from the AgentGym benchmark, demonstrating significant improvements over prompting baselines and even outperforming Group Relative Policy Optimization (GRPO) as we scale the test-time compute of our method. AMC demonstrates the feasibility of performing principled RL-style optimization of black-box LLM agents. Code is available at https://github.com/layer6ai-labs/Agentic-Monte-Carlo
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。