评测大模型生成复杂C++单元测试能力，发现微调后性能提升70%以上。
CPP-UT-Bench: Can LLMs Write Complex Unit Tests in C++?
- 构建包含2653组代码-测试对的C++单元测试基准数据集
- 微调后的模型在9项实验中平均性能超越基础模型70%以上
- 适用于评估和改进AI在真实工业级C++代码上的测试生成能力
我们提出了CPP-UT-Bench，一个用于衡量大语言模型（LLM）生成C++单元测试能力的基准数据集。该数据集涵盖来自14个开源C++项目、横跨机器学习、软件测试、解析、标准输入输出、数据工程、日志、表达式求值、键值存储及服务器协议等九个不同领域的2,653组{代码, 单元测试}对。通过上下文学习、参数高效微调（PEFT）和全参数微调的大量实验，验证了该数据集的有效性。我们还讨论了数据集构建中的挑战，并从实验中获得关键洞察。除数据集与编译代码外，我们还公开了微调后的模型权重，供后续研究使用。在十项实验中，九项显示微调模型相比基线模型平均性能提升超过70%。
原文摘要 · Abstract (English)
We introduce CPP-UT-Bench, a benchmark dataset to measure C++ unit test generation capability of a large language model (LLM). CPP-UT-Bench aims to reflect a broad and diverse set of C++ codebases found in the real world. The dataset includes 2,653 {code, unit test} pairs drawn from 14 different opensource C++ codebases spanned across nine diverse domains including machine learning, software testing, parsing, standard input-output, data engineering, logging, complete expression evaluation, key value storage, and server protocols. We demonstrated the effectiveness of CPP-UT-Bench as a benchmark dataset through extensive experiments in in-context learning, parameter-efficient fine-tuning (PEFT), and full-parameter fine-tuning. We also discussed the challenges of the dataset compilation and insights we learned from in-context learning and fine-tuning experiments. Besides the CPP-UT-Bench dataset and data compilation code, we are also offering the fine-tuned model weights for further research. For nine out of ten experiments, our fine-tuned LLMs outperformed the corresponding base models by an average of more than 70%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。