arXiv:2608.16798cs.CLcs.AI2026-08

提出黑盒强化学习框架,让智能体在复杂协作系统中稳定高效训练。

ClawGym II: Exploring Black-Box RL on Agent Harness

  • 构建沙箱环境实现并发任务隔离,通过代理捕获模型调用
  • 用前缀树重构多轮交互轨迹,支持基于和无基线的策略优化
  • 支持异构协作系统统一训练,提升长时任务表现

智能体协作系统显著提升了长时任务的表现,但通过复杂协作系统进行强化学习仍面临根本挑战。本文提出一种统一的黑盒强化学习框架,实现通用智能体在复杂协作系统中的稳定、可扩展优化。首先,构建基于沙箱的执行架构,将任务环境与协作系统隔离于临时沙箱中,支持大规模并发推演。其次,将策略优化与不透明协作执行解耦,在模型边界部署服务代理以捕获模型调用。为重建多轮轨迹并提升训练效率,将捕获调用组织成前缀树,并适配基于评论家的PPO与无评论家的GRPO算法以优化恢复的树结构。同时保持训练-推理一致性。最后引入混合协作训练,使单一模型可被异构协作系统联合优化。使用Qwen3-30A3B,在ClawGym-Bench上,通过OpenClaw和Claude Code分别实现Pass@1提升9.98和14.81点,且在200-400次优化步骤中保持稳定。该框架在更复杂的JobBench和OfficeQA任务中也取得一致提升。整体实现了通过黑盒协作系统对通用智能体的有效、稳定、可扩展优化,支持跨异构执行系统的统一训练。

原文摘要 · Abstract (English)

Agent harnesses have substantially improved performance on long-horizon tasks by coordinating agent interactions with the environment. However, reinforcement learning through complex harnesses remains largely unexplored, as scaling such training to long-horizon agent tasks introduces fundamental challenges. In this work, we present a unified black-box RL framework for stable and scalable optimization of general agents through complex harnesses. Concretely, we first build a sandbox-based execution infrastructure that isolates task environments and harnesses within temporary sandboxes for large-scale concurrent rollouts. We then decouple policy optimization from opaque harness execution and place a serving proxy at the model boundary to capture model calls. To reconstruct multi-turn trajectories and improve training efficiency, we organize the captured calls into prefix trees and further adapt both critic-based PPO and critic-free GRPO to optimize over the recovered tree structure. Meanwhile, we maintain training-inference consistency throughout the optimization process. Finally, we introduce mix-harness training, allowing a single model to be jointly optimized by heterogeneous harnesses. With Qwen3-30A3B, black-box RL improves Pass@1 on ClawGym-Bench by 9.98 and 14.81 points through OpenClaw and Claude Code, respectively, while remaining stable over 200-400 optimization steps. Moreover, the framework yields consistent gains on more challenging tasks such as JobBench and OfficeQA. Overall, our framework enables effective, stable, and scalable optimization of general agents through black-box harnesses, supporting unified training across heterogeneous execution systems.

强化学习智能体协作黑盒优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。