多规则训练的代码智能体，真正提升性能的是评估环境而非训练策略。
What Does Multi-Harness RL Learn? Credit Assignment and Portability in Coding Agents
- 用同一任务记录在不同评测环境间对比，分离训练策略影响。
- 评测环境差异导致解题率从2.14%升至9.27%，提升4.3倍。
- 跨环境信用分配不带来更强泛化能力，仅适应当前评测配置。
强化学习智能体在完整执行环境中运行日益普遍，多环境训练策略包含两个关键选择：暴露策略于多个环境，以及在相对优势组内比较奖励。本文聚焦于第二个选择，在代码生成领域进行隔离实验。基于一个预训练的Qwen3-8B模型，重复播放来自Aider、OpenHands、Qwen Code和SWE-agent的相同冻结任务-环境记录，采用相同的更新次数，在两种分组优化规则下评估：组内（每任务-环境对独立分组）与跨组（同一任务内环境合并分组）。使用密封的SWE-bench Verified评测器，在四个源环境及一个未参与训练的最小环境上评分。结果显示，评测环境是主导变量：24,000次封闭评估中，其使平均解题率从2.14%提升至9.27%（提升4.3倍），而训练策略仅带来1.16倍变化。分组规则无显著差异：跨组减组内为+0.25个百分点（95%置信区间[-0.48, +1.02]），三次训练种子的汇总结果为+0.16（[-0.41, +0.72]），各种子估计值符号可变。两规则各自内部种子范围达0.42–0.45个百分点，超过两者间差异。两者最大收益均集中于同一源环境。跨组优势能准确识别生成环境（分类准确率高出基准4.48个百分点），而组内则无法。尽管如此，两规则在未见环境中的最终得分与动作分布仍一致。在策略上重新收集一半训练数据也不改变该结论。跨环境信用分配仅实现配置适应，未带来更优可迁移能力。多环境强化学习报告应明确分组边界，并在未见环境下测试。
原文摘要 · Abstract (English)
Agent reinforcement learning (RL) increasingly runs through full execution harnesses, and a multi-harness recipe mixes two choices: exposing the policy to several harnesses, and comparing their rewards inside one relative-advantage group. We isolate the second choice in repository-level coding. From one Qwen3-8B supervised warm start we replay the same frozen task-harness records from Aider, OpenHands, Qwen Code, and SWE-agent, with the same number of updates, under two rules for group-relative policy optimization (GRPO), Within (one group per task-harness pair) and Cross (harnesses pooled within a task), and score every checkpoint with a sealed SWE-bench Verified oracle on four source harnesses and a minimal harness held out of training. The evaluation harness is the dominant variable: across 24,000 sealed evaluations it moves the mean solve rate from 2.14\% to 9.27\%, a factor of 4.3, where the training recipe moves it by 1.16. The grouping rule is not. On the held-out harness, Cross minus Within is +0.25 pp, 95\% confidence interval [-0.48, +1.02], at eight attempts per task, and +0.16 [-0.41, +0.72] pooled over three training seeds whose individual estimates change sign. Each rule's own seed range, 0.42 to 0.45 pp, exceeds the difference between them. Both rules place their largest gains on the same source harness. The pooled advantage carries the harness: an out-of-fold classifier recovers the generating harness from Cross's advantage +4.48 pp above the shuffled-label baseline and from Within's not at all, and the two rules still reach the same held-out score and action distribution inside each harness. Re-collecting half the training data on-policy does not change this. Cross-harness credit yields configuration adaptation and no more portable capability than within-harness credit. Multi-harness RL reports should state the grouping boundary and test under an unseen harness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。