首个面向多文件代码的LLM单元测试生成基准，揭示模型在真实项目中的短板。
MultiFileTest: A Multi-File-Level LLM Unit Test Generation Benchmark and Impact of Error Fixing Mechanisms
- 构建跨语言多文件级测试生成评测集，覆盖Python/Java/JavaScript
- 多数前沿模型在多文件场景下表现平庸，存在可执行性与连锁错误
- 验证手动与自修复机制能显著提升生成质量，适合研究测试生成与纠错
单元测试生成已成为大型语言模型（LLM）的重要应用方向。然而，现有评估基准多聚焦于函数或类级别（单文件）代码，难以反映实际多文件代码库的复杂性。为此，我们提出MultiFileTest——一个涵盖Python、Java和JavaScript的多文件级单元测试生成基准，每个语言包含20个中等规模且高质量的项目。我们在11个前沿LLM上进行评估，结果表明大多数模型在MultiFileTest上的表现仅处于中等水平，凸显其挑战性。进一步的错误分析发现，即使是Gemini-3.0-Pro等先进模型，仍存在可执行性错误与连锁错误等基础但关键的问题。基于此，我们还评估了人工修复与自修复机制下的模型表现，以考察错误修复能力对生成效果的潜在提升。相关代码与数据集已开源。
原文摘要 · Abstract (English)
Unit test generation has become a promising and important Large Language Model (LLM) use case. However, existing evaluation benchmarks for LLM unit test generation focus on function- or class-level code (single-file) rather than more practical and challenging multi-file-level codebases. To address such a limitation, we propose MultiFileTest, a multi-file-level benchmark for unit test generation covering Python, Java, and JavaScript. MultiFileTest features 20 moderate-sized and high-quality projects per language. We evaluate eleven frontier LLMs on MultiFileTest, and the results show that most frontier LLMs tested exhibit moderate performance on MultiFileTest, highlighting the difficulty of MultiFileTest. We also conduct a thorough error analysis, which shows that even advanced LLMs, such as Gemini-3.0-Pro, exhibit basic yet critical errors, including executability and cascade errors. Motivated by this observation, we further evaluate all frontier LLMs under manual error-fixing and self-error-fixing scenarios to assess their potential when equipped with error-fixing mechanisms. Our code and dataset is available at \href{https://github.com/YiboWANG214/ProjectTest}{MultiFileTest}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。