对比人类与智能体在机器学习开发中的规划差异,揭示其效率差距根源。
TraceML: An Empirical Analysis of Human-Agent Planning in Machine Learning Development

- 构建统一版本框架追踪人类与智能体的开发过程
- 人类高频切换任务且重用旧方案,智能体困于单一循环
- 简化规划指令可提升性能但无法改变行为模式
大型语言模型能解决孤立编程问题,但在自主机器学习开发中仍远逊于人类——需在数小时反馈中迭代数据管道、模型和验证策略。现有结果导向基准仅评价最终提交,忽略开发过程。我们提出TraceML,将人类与智能体在相同竞赛中工作流程对齐至同一版本级架构:涵盖134场竞赛中4,465条人类轨迹,其中7场也由两个智能体框架完成,共得430条配对的人类轨迹与207条智能体轨迹。每条代码版本记录得分、时间戳、操作标签、意图、修改规模及得分影响。分析显示,专家交替进行数据、验证、模型调整与集成,并重新启用曾搁置方法;而各智能体框架则陷入狭窄循环:Codex持续调整集成权重,MLEvolve原地突变模型,均无法如人类般灵活转向或重启旧路径。从人类实践中提炼的简短规划提示虽使行为更接近人类并提升分数,但整体执行模式仍保持智能体特征:仅指令部分可被弥补。数据集、标注体系、标签者与提取工具已公开于https://huggingface.co/datasets/jerryyan/TraceML。
原文摘要 · Abstract (English)
Large language models write correct code for isolated problems but remain far weaker at autonomous machine-learning development, where an agent must revise data pipelines, models, and validation over hours of feedback, and on most competitions still finishes below strong human competitors. Outcome-based benchmarks record this gap but not its cause, because they grade the final submission and discard the development process behind it. We introduce TraceML, which pairs human and agent work on the same competitions under one version-level schema: 4,465 human Kaggle trajectories across 134 competitions, seven of which are also worked by two agent scaffolds, giving 430 paired human and 207 agent trajectories. Every code version carries its score, its timestamp, and labels for the action taken, its intent, the edit size, and the score effect. Read this way, the gap becomes concrete. Experts alternate data work, validation, model changes, and ensembling, and return to approaches they had set aside. Each agent scaffold instead collapses into a narrow loop: Codex spends its steps re-weighting ensembles and tuning submissions, MLEvolve mutates its model in place, and neither pivots at the human rate nor reopens abandoned work. A short planning prompt distilled from human practice moves the behaviors it names toward the human profile and lifts scores, but the effort profile stays agent-shaped: instruction closes only the part of the gap that reduces to instructions. We release the corpus, the schema, the labelers, and the extraction pipeline at https://huggingface.co/datasets/jerryyan/TraceML.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。