发现并检测代码生成任务描述中的缺陷,提升大模型输出准确性。
Defective Task Descriptions in LLM-Based Code Generation: Detection and Analysis

- 用轻量小模型微调构建缺陷检测器SpecValidator。
- 在三类缺陷上实现F1=0.804,显著优于GPT-5-mini等模型。
- 可识别未见过的不完整缺陷,适合提升真实场景代码生成质量。
大型语言模型广泛用于代码生成,但依赖于任务描述详尽且结构良好的隐含假设。实践中用户常提供有缺陷的描述,严重影响代码正确性。为此,我们提出SpecValidator,一个基于小模型参数高效微调的轻量级分类器,用于自动检测任务描述缺陷。我们在三个基准数据集上评估了词汇模糊、描述不足和语法格式三类缺陷,结果表明SpecValidator在F1=0.804、MCC=0.745,显著优于GPT-5-mini(F1=0.469,MCC=0.281)和Claude Sonnet 4(F1=0.518,MCC=0.359)。更重要的是,分析显示SpecValidator能泛化至未见问题,检测原始基准中未知的描述不足缺陷。研究还发现,大模型对缺陷的鲁棒性主要取决于缺陷类型与描述特征,而非模型规模,其中描述不足最为严重。具备更丰富上下文信息的基准(如LiveCodeBench)表现出更强抗干扰能力,凸显结构化描述对可靠代码生成的重要性。
原文摘要 · Abstract (English)
Large language models are widely used for code generation, yet they rely on an implicit assumption that the task descriptions are sufficiently detailed and well-formed. However, in practice, users may provide defective descriptions, which can have a strong effect on code correctness. To address this issue, we develop SpecValidator, a lightweight classifier based on a small model that has been parameter-efficiently finetuned, to automatically detect task description defects. We evaluate SpecValidator on three types of defects, Lexical Vagueness, Under-Specification and Syntax-Formatting on 3 benchmarks with task descriptions of varying structure and complexity. Our results show that SpecValidator achieves defect detection of F1 = 0.804 and MCC = 0.745, significantly outperforming GPT-5-mini (F1 = 0.469 and MCC = 0.281) and Claude Sonnet 4 (F1 = 0.518 and MCC = 0.359). Perhaps more importantly, our analysis indicates that SpecValidator can generalize to unseen issues and detect unknown Under-Specification defects in the original (real) descriptions of the benchmarks used. Our results also show that the robustness of LLMs in task description defects depends primarily on the type of defect and the characteristics of the task description, rather than the capacity of the model, with Under-Specification defects being the most severe. We further found that benchmarks with richer contextual grounding, such as LiveCodeBench, exhibit substantially greater resilience, highlighting the importance of structured task descriptions for reliable LLM-based code generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。