首个面向多库代码调试的基准测试，揭示大模型在此场景下的短板。
MLDebugging: Towards Benchmarking Code Debugging Across Multi-Library Scenarios
- 构建涵盖126个库的多库调试基准MLDebugging
- 发现主流大模型在跨库调试中准确率仍很低
- 适合研究大模型代码理解与调试能力的学者
代码调试是软件工程中的关键任务，近年来受到广泛关注。尽管大语言模型（LLMs）取得显著进展，但现有研究仍集中于无库或单库场景，忽略了真实应用中复杂的多库环境。为此，我们首次提出MLDebugging（多库调试）基准，用于评估多库Python代码中的调试挑战。该基准包含126个不同的Python库，覆盖多种多库代码问题，分为七类。我们对主流开源和闭源大模型进行了全面评估，结果表明当前大模型在跨库调试任务中仍表现不佳。本工作旨在揭示大模型在多库调试场景中的潜力，并为未来研究提供参考。
原文摘要 · Abstract (English)
Code debugging is a crucial task in software engineering, which attracts increasing attention. While remarkable success has been made in the era of large language models (LLMs), current research still focuses on the simple no-library or single-library setting, ignoring the complex multi-library scenario in real-world applications. To address this limitation, we make the first attempt to introduce MLDebugging (Multi-Library Debugging), a comprehensive benchmark designed to assess debugging challenges within multi-library Python code. Specifically, MLDebugging encompasses 126 distinct Python libraries, covering a wide range of multi-library code issues, categorized into seven distinct types. Furthermore, we conduct a thorough evaluation of MLDebugging using both mainstream open-source and closed-source LLMs and highlight that current LLMs still struggle to correctly perform code debugging across multi-library scenarios. We hope this work can uncover the potential of LLMs in multi-library debugging scenario and offer insights for future research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。