让数据科学自动化中的任务框架显式化，提升可复现性与可比性。
DS-Lighting: Making Agent Harnesses Explicit for Data-Science Automation
- 将任务框架拆分为数据、流程、执行、评估四层，支持可复用的程序化设计
- 在多个基准上实现更稳定的结果，减少系统级错误发生率
- 适合希望提升自动化实验可复现性的研究人员与工程师
大型语言模型（LLM）代理在自动化数据科学工作流方面展现出潜力，但其端到端性能高度依赖于任务框架——该框架负责任务表示、执行状态管理、输出约束和评估反馈。现有数据科学代理常使框架隐式存在，导致结果难以在异构任务间复现、比较和归因。我们提出 DS-Lighting，一个统一的框架工具包，使数据科学自动化的框架设计显式化。DS-Lighting 将框架分解为四个可复用层：数据、工作流、执行和评估，并将不同代理表示为可执行的操作符程序，支持预定义流水线与自适应搜索。我们进一步将多个开源数据科学基准整合为 MLE-Bench 风格的任务格式，实现共享接口、沙箱运行环境与统一评估协议下的可控比较。跨代理、框架、模型及消融实验表明，显式框架设计显著提升可复现性、可比性和可靠性，同时降低端到端数据科学工作流中不必要的系统级失败。代码已公开于 https://github.com/usail-hkust/dslighting。
原文摘要 · Abstract (English)
Large Language Model (LLM) agents have shown promise for automating data-science workflows, yet their end-to-end performance depends critically on the agent harness that represents tasks, manages execution state, constrains output artifacts, and provides evaluation feedback. Existing data-science agents often leave this harness implicit, making results difficult to reproduce, compare, and attribute across heterogeneous tasks. We introduce DS-Lighting, a unified harness toolkit that makes harness design explicit for data-science automation. DS-Lighting decomposes the harness into four reusable layers: data, workflow, execution, and evaluation, and represents diverse agents as executable operator programs that support both predefined pipelines and adaptive search. We further integrate multiple open-source data-science benchmarks into an MLE-Bench-style task format, enabling controlled comparison under a shared task interface, sandboxed runtime, and metric protocol. Experiments across agents, harnesses, models, and ablations show that explicit harness design improves reproducibility, comparability, and reliability, while reducing avoidable system-level failures in end-to-end data-science workflows. Our code is available at https://github.com/usail-hkust/dslighting
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。