编码智能体强化学习中，执行环境差异导致效率差距超百倍，需优化基础设施。
The Rollout Infrastructure Tax in Coding-Agent Reinforcement Learning
- 对比四种执行环境：单容器、托管沙箱、K8s容器、云虚拟机
- 冷启动延迟最高差110倍，百万轨迹预估工时差1.8倍
- 适合关注训练效率的强化学习系统设计者
编码智能体强化学习将执行基础设施视为背景实现细节，尽管其依赖大量交互式软件部署。这造成重大遗漏：测量基础设施开销可揭示强化学习训练后实用效率提升潜力，微小的每轮节省在大规模下累积显著。我们对四种执行底座进行对比研究：单容器、托管沙箱、Kubernetes编排容器与云虚拟机。结果显示，冷启动延迟最高相差110倍，一百万条150步轨迹的预估工时差异达1.8倍。结果表明，未来编码智能体强化学习系统应将执行底座优化纳入训练体系本身，而非仅作为部署工具。
原文摘要 · Abstract (English)
Coding-agent reinforcement learning treats execution infrastructure as a background implementation detail, despite relying on large numbers of interactive software rollouts. This is a missed opportunity: measuring infrastructure overhead can reveal practical efficiency gains for RL post-training, where small per-rollout savings compound at scale. We present a comparative study of four execution substrates: single containers, hosted sandboxes, Kubernetes-orchestrated containers, and cloud virtual machines. We find up to $110\times$ variation in cold-start latency and a $1.8\times$ spread in projected worker-hours for one million 150-step trajectories. Our results suggest that future coding-agent RL systems should optimize execution substrates as part of the training system itself, not merely as deployment plumbing.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。