评测AI代理在真实机器学习工程任务中的表现，发现顶尖模型仅在17%任务中达到人类青铜水平。
MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering
- 构建75个真实Kaggle任务组成的基准，覆盖数据准备、模型训练等全流程工程
- 最优模型（o1-preview+AIDE）在16.9%任务中达到人类青铜级表现
- 开源代码，助力研究AI在机器学习工程中的实际能力边界
我们提出MLE-bench，一个用于评估人工智能代理在机器学习工程领域表现的基准。为此，我们从Kaggle收集了75个与机器学习工程相关的竞赛，涵盖训练模型、数据集准备和实验运行等真实场景任务。通过Kaggle公开排行榜建立人类基线。使用开源代理框架评估多个前沿语言模型，发现最佳配置（OpenAI o1-preview搭配AIDE框架）在16.9%的竞赛中至少达到Kaggle青铜奖水平。此外，我们研究了资源扩展对代理性能的影响及预训练数据污染的后果。我们已将基准代码开源（github.com/openai/mle-bench/），以促进对AI代理机器学习工程能力的后续研究。
原文摘要 · Abstract (English)
We introduce MLE-bench, a benchmark for measuring how well AI agents perform at machine learning engineering. To this end, we curate 75 ML engineering-related competitions from Kaggle, creating a diverse set of challenging tasks that test real-world ML engineering skills such as training models, preparing datasets, and running experiments. We establish human baselines for each competition using Kaggle's publicly available leaderboards. We use open-source agent scaffolds to evaluate several frontier language models on our benchmark, finding that the best-performing setup--OpenAI's o1-preview with AIDE scaffolding--achieves at least the level of a Kaggle bronze medal in 16.9% of competitions. In addition to our main results, we investigate various forms of resource scaling for AI agents and the impact of contamination from pre-training. We open-source our benchmark code (github.com/openai/mle-bench/) to facilitate future research in understanding the ML engineering capabilities of AI agents.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。