arXiv:2605.07024cs.LGcs.AI2026-05被引 2

构建多语言代码幻觉检测基准,揭示大模型在填空任务中的系统性错误。

Delulu: A Verified Multi-Lingual Benchmark for Code Hallucination Detection in Fill-in-the-Middle Tasks

论文配图:Delulu: A Verified Multi-Lingual Benchmark for Code Hallucination Detection in Fill-in-the-Middle Tasks
图 1 · 摘自论文原文
  • 通过对抗生成与多模型验证,构建1951个跨语言代码幻觉样本。
  • 最强模型仅84.5%准确率,所有模型均存在幻觉对齐输出。
  • 适合评估代码生成模型可靠性,尤其关注运行时错误风险。

代码生成的大语言模型在填空任务中常产生幻觉——即看似合理但实际错误的补全,如虚构的API方法、无效参数、未定义变量或不存在的导入。这些错误通过表面审查却导致运行时异常。我们提出Delulu,一个经验证的多语言基准,包含1951个样本,覆盖7种语言和4类幻觉。样本通过对抗式流程生成:前沿大模型生成合理幻觉,四个不同判别模型评估,基于嵌入聚类挖掘更难样本,自包含Docker容器验证正确补全可编译而幻觉版本产生预期运行时错误,最终由人工专家审查剔除偏倚或过于简单样本。我们评估了来自五个家族的11个开源权重FIM模型(0.5B-32B参数):六组Qwen2.5-Coder扩展系列,以及跨家族系列(CodeLlama、DeepSeek-Coder-V2、StarCoder2)。最强模型仅达84.5% pass@1,无一模型编辑相似度超0.77,且每个家族均在非平凡样本上生成幻觉对齐输出,证明该难度为任务固有而非家族特异性。基准、容器与评估框架已开源:https://github.com/microsoft/delulu。

原文摘要 · Abstract (English)

Large Language Models for code generation frequently produce hallucinations in Fill-in-the-Middle (FIM) tasks -- plausible but incorrect completions such as invented API methods, invalid parameters, undefined variables, or non-existent imports. These failures pass superficial review yet introduce runtime errors. We introduce Delulu, a verified multi-lingual benchmark of 1,951 FIM samples across 7 languages and 4 hallucination types. Samples are curated through an adversarial pipeline: a frontier LLM generates plausible hallucinations, four diverse judge models evaluate them, embedding-based clustering mines progressively harder examples, self-contained Docker containers verify that golden completions compile while hallucinated variants produce the expected runtime error, and a final human-expert review removes any remaining biased or trivially decidable samples. We evaluate 11 open-weight FIM models from five families spanning 0.5B-32B parameters: a six-point Qwen2.5-Coder scaling slate, plus a cross-family slate (CodeLlama, DeepSeek-Coder-V2, StarCoder2). The strongest model reaches only 84.5% pass@1, no family exceeds 0.77 Edit Similarity, and every family produces hallucination-aligned completions on a non-trivial share of samples, confirming that the difficulty exposed by Delulu is task-intrinsic rather than family-specific. We release the benchmark, containers, and evaluation framework at https://github.com/microsoft/delulu.

代码生成幻觉检测多语言评测基准

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。