首个评估大模型多步多错误调试能力的基准
Why Stop at One Error? Benchmarking LLMs as Data Science Code Debuggers for Multi-Hop and Multi-Bug Errors
- 构建真实数据科学代码的多步推理与多错误检测任务
- 含1117个样本、741对因果错误链，支持运行时错误诊断
- 适合研究大模型代码修复与复杂逻辑推理的学者使用
大语言模型正改变软件开发，但现有代码生成与修复评测主要聚焦简单单错误场景的语法和功能正确性。模型在复杂数据科学代码中自主发现并修复运行时逻辑错误的能力尚未被充分探索。为此，我们提出DSDBench：首个系统评估大模型在数据科学代码调试中多步错误追踪与多错误检测能力的基准。该基准改编自现有数据科学任务基准（如DABench和MatPlotBench），包含自动合成的多步、多错误代码片段，覆盖真实数据科学调试任务。DSDBench包含1,117个标注样本，含741对因果错误链及运行时错误信息。对主流大模型的评估显示显著性能差距，凸显其在数据科学代码逻辑错误调试中的挑战。该基准可推动大模型调试与推理能力提升，助力未来更可靠的AI辅助数据科学。项目已开源：github.com/KevinCL16/DSDBench。
原文摘要 · Abstract (English)
LLMs are transforming software development, yet current code generation and code repair benchmarks mainly assess syntactic and functional correctness in simple, single-error cases. LLMs' capabilities to autonomously find and fix runtime logical errors in complex data science code remain largely unexplored. To address this gap, we introduce DSDBench: the Data Science Debugging Benchmark, the first benchmark for systematic evaluation of LLMs on multi-hop error tracing and multi-bug detection in data science code debugging. DSDBench adapts datasets from existing data science task benchmarks, such as DABench and MatPlotBench, featuring realistic data science debugging tasks with automatically synthesized multi-hop, multi-bug code snippets. DSDBench includes 1,117 annotated samples with 741 cause-effect error pairs and runtime error messages. Evaluations of state-of-the-art LLMs on DSDBench show significant performance gaps, highlighting challenges in debugging logical runtime errors in data science code. DSDBench offers a crucial resource to evaluate and improve LLMs' debugging and reasoning capabilities, enabling more reliable AI-assisted data science in the future. DSDBench is publicly available at github.com/KevinCL16/DSDBench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。