评测10个开源大模型在表格机器学习任务中的自动编码能力
TML-Bench: Benchmark for Data Science Agents on Tabular ML Tasks
- 构建TML-Bench基准，评估模型在4个Kaggle任务上的端到端表现
- 时间越长模型表现越好，240秒到1200秒间平均性能提升显著
- 适合关注自动化数据科学工具的开发者与研究者
自主编码代理能在Kaggle风格任务中快速生成强大的表格机器学习基线。其实际价值取决于在时间限制下端到端的正确性和可靠性。本文提出TML-Bench，一个面向表格机器学习任务的数据科学代理基准。评估了10个开源LLM在4个Kaggle竞赛和3种时间预算（240秒、600秒、1200秒）下的表现，每项任务与预算运行5次。一次运行成功需生成有效提交，并在不可见的私有测试集上获得评分。报告了中位性能、成功率及运行间变异性。MiniMax-M2.1在所有四个竞赛中取得最优综合得分。平均性能随时间预算增大而提升。部分模型在当前运行次数下缩放表现不稳定。代码与材料已公开于https://github.com/MykolaPinchuk/TML-bench/tree/master。
原文摘要 · Abstract (English)
Autonomous coding agents can produce strong tabular baselines quickly on Kaggle-style tasks. Practical value depends on end-to-end correctness and reliability under time limits. This paper introduces TML-Bench, a tabular benchmark for data science agents on Kaggle-style tasks. This paper evaluates 10 OSS LLMs on four Kaggle competitions and three time budgets (240s, 600s, and 1200s). Each model is run five times per task and budget. A run is successful if it produces a valid submission and a private-holdout score on hidden labels that are not accessible to the agent. This paper reports median performance, success rates, and run-to-run variability. MiniMax-M2.1 model achieves the best aggregate performance score on all four competitions under the paper's primary aggregation. Average performance improves with larger time budgets. Scaling is noisy for some individual models at the current run count. Code and materials are available at https://github.com/MykolaPinchuk/TML-bench/tree/master.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。