arXiv:2604.04872cs.CLcs.LG2026-04被引 8

用小数据生成多样仿真环境,让机器学习工程代理高效训练。

Synthetic Sandbox for Training Machine Learning Engineering Agents

  • 用少量种子任务生成微型合成环境,保持真实复杂性
  • 执行时间减少13倍以上,首次实现大规模在线策略强化学习
  • 在多个模型上提升显著,且能泛化到新框架

随着大语言模型代理从软件工程迈向机器学习工程,验证其行为成本急剧上升:软件工程可通过快速单元测试验证,而机器学习工程需在每一步策略迭代中运行完整流程——包括数据预处理、模型训练和指标评估,使用大规模数据集,导致轨迹级在线强化学习极为缓慢。现有方法退回到监督微调或离线代理奖励,牺牲了在线强化学习的探索与泛化优势。我们发现沙盒数据规模是主要瓶颈。基于此,提出 SandMLE 框架,从少量种子任务生成多样化、可验证的合成机器学习工程环境,保留真实问题的结构与技术复杂度,同时将每个任务的数据量控制在50-200样本的微尺度。实验表明,SandMLE 将执行时间降低超过13倍,首次实现在机器学习工程领域的大规模轨迹级在线强化学习。在 MLE-bench-lite 上,相比监督微调基线,Qwen3-8B、14B 和 30B-A3B 模型的相对奖牌率提升达20.3%至66.9%。此外,训练出的策略能泛化至未见过的智能体架构,在 MLE-Dojo 上人类评分(HumanRank)最高提升32.4%。

原文摘要 · Abstract (English)

As large language model agents advance beyond software engineering (SWE) tasks toward machine learning engineering (MLE), verifying agent behavior becomes orders of magnitude more expensive: while SWE tasks can be verified via fast-executing unit tests, MLE verification requires running full ML pipelines -- data preprocessing, model training, and metric evaluation -- on large datasets at each rollout step, rendering trajectory-wise on-policy reinforcement learning (RL) prohibitively slow. Existing approaches retreat to supervised fine-tuning (SFT) or offline proxy rewards, sacrificing the exploration and generalization benefits of on-policy RL. We observe that sandbox data size is the primary source of this bottleneck. Based on this insight, we introduce SandMLE, a multi-agent framework that generates diverse, verifiable synthetic MLE environments from a small number of seed tasks, preserving the structural and technical complexity of real-world problems while constraining datasets to micro-scale (each task is paired with only 50-200 training samples). Through extensive experiments, we show that SandMLE reduces execution time by over 13 times, enabling large-scale, on-policy trajectory-wise RL for the first time in the MLE domain. On MLE-bench-lite, SandMLE yields significant gains over SFT baselines across Qwen3-8B, 14B, and 30B-A3B, with relative medal rate improvements ranging from 20.3% to 66.9%. Furthermore, the trained policy generalizes across unseen agentic scaffolds, achieving up to 32.4% better HumanRank score on MLE-Dojo.

机器学习工程强化学习仿真环境代理训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。