用规范文档替代有缺陷代码，可有效减少大模型生成测试时的误导。
Evaluating and Mitigating the Misguidance Effect of Buggy Code in LLM-Generated Unit Tests
- 用规范文档替换有缺陷代码作为提示，避免模型被错误行为引导。
- 相比原方法，误导性测试减少67%，发现缺陷的有效测试增加3.2倍。
- 适合需要高质量自动化测试的开发者，尤其在修复复杂代码时实用。
尽管大语言模型（LLMs）在自动化单元测试生成方面展现出巨大潜力，但近期研究指出，当模型被提供有缺陷的代码作为提示时，生成测试的质量会受到负面影响。本文提出一种新度量指标，定量评估‘误导效应’——即有缺陷代码会引导模型生成验证错误行为的测试，而非暴露缺陷。分析显示，使用有缺陷代码作为提示会产生双重负面影响：误导性测试显著增加，同时抑制了有效查错测试的生成。从模型内部视角进一步证实，有缺陷代码会改变模型对测试断言的偏好，使其更倾向于支持错误行为。为应对该问题，本文引入并验证了一种基于规范文档的单元测试生成范式：将待测代码替换为模型生成的规范文档作为提示。结果表明，该范式能有效降低误导性测试比例，大幅提升有效测试数量，并提升多轮反馈式测试生成效率，且适用于有缺陷和无缺陷代码。总体而言，基于规范的提示是缓解有缺陷代码误导性的有效策略。
原文摘要 · Abstract (English)
While Large Language Models (LLMs) show great promise for automating unit test generation, recent studies suggest that the quality of generated tests can be negatively impacted when models are prompted with buggy code. This paper presents a new metric to quantitatively measure the "misguidance effect," a phenomenon where buggy code steers LLMs toward generating tests that validate its erroneous behavior rather than expose it. Our analysis reveals that prompting LLMs with buggy code has a severe, twofold impact: it significantly increases "misguided tests" that assert incorrect behavior while simultaneously suppressing the generation of effective, bug-finding tests. We further corroborate this effect from a model-internal perspective, showing that buggy code skews LLMs' preference toward tests that assert the same erroneous behavior. To counter this, we introduce and validate a specification-based unit test generation paradigm that replaces the code under test in the prompt with an LLM-generated specification docstring. Our results show that this paradigm effectively reduces misguided tests while substantially increasing effective tests, improves multi-round, feedback-driven test generation pipelines, and remains applicable to both buggy and bug-free code. Overall, these results suggest that specification-based prompting is a promising strategy for mitigating misguidance from buggy code in LLM-generated unit tests.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。