用多语言和温度控制生成测试，统一矛盾结果提升质量。
Unify and Triumph: Polyglot, Diverse, and Self-Consistent Generation of Unit Tests with LLMs
- 跨语言与多采样生成测试，再统一消除矛盾。
- 测试通过率、覆盖率最高提升9.01%，突变分数提升11.23%。
- 适合低性能语言或需高鲁棒性测试的开发者使用。
基于大语言模型（LLM）的测试生成在软件工程中备受关注，但多数研究仅评估单一语言单次生成能力，未能利用模型多样性提升测试鲁棒性。本文提出PolyTest，通过多语言零温生成与高温度多样采样，系统性地融合两种互补策略：(1) 跨语言生成——在五种语言（Java、C、Python、JavaScript、CSV）中零温生成测试并统一；(2) 多样采样——同一语言高温生成多组测试后统一。关键洞察是，不同语言与生成次数下，模型常输出相同输入却不同预期结果的矛盾测试。PolyTest通过统一机制实现自一致性，显著提升测试质量。实验在Llama3-70B、GPT-4o与GPT-3.5上使用EvalPlus进行，结果表明该方法在所有指标上均优于单次生成，测试数量、通过率、语句/分支覆盖率最高提升9.01%，突变分数提升11.23%。最终，PolyTest在生成效率、通过率及突变分数上均优于Pynguin。
原文摘要 · Abstract (English)
Large language model (LLM)-based test generation has gained attention in software engineering, yet most studies evaluate LLMs' ability to generate unit tests in a single attempt for a given language, missing the opportunity to leverage LLM diversity for more robust testing. This paper introduces PolyTest, a novel approach that enhances test generation by exploiting polyglot and temperature-controlled diversity. PolyTest systematically leverages these properties in two complementary ways: (1) Cross-lingual test generation, where tests are generated in multiple languages at zero temperature and then unified; (2) Diverse test sampling, where multiple test sets are generated within the same language at a higher temperature before unification. A key insight is that LLMs can generate diverse yet contradicting tests -- same input, different expected outputs -- across languages and generations. PolyTest mitigates inconsistencies by unifying test sets, fostering self-consistency and improving overall test quality. Unlike single-language or single-attempt approaches, PolyTest enhances testing without requiring on-the-fly execution, making it particularly beneficial for weaker-performing languages. We evaluate PolyTest on Llama3-70B, GPT-4o, and GPT-3.5 using EvalPlus, generating tests in five languages (Java, C, Python, JavaScript, and a CSV-based format) at temperature 0 and sampling multiple sets at temperature 1. We observe that LLMs frequently generate contradicting tests across settings, and that PolyTest significantly improves test quality across all considered metrics -- number of tests, passing rate, statement/branch coverage (up to +9.01%), and mutation score (up to +11.23%). Finally, PolyTest outperforms Pynguin in test generation, passing rate, and mutation score.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。