通过追踪代码实体检测生成摘要中的幻觉，准确率超73%。
ETF: An Entity Tracing Framework for Hallucination Detection in Code Summaries
- 用静态分析识别代码实体，再由大模型验证其意图。
- 在1万条数据上实现73%的F1分数，有效定位错误位置。
- 适合关注代码生成质量与可信度的研究者使用。
大语言模型在自然语言与代码理解方面取得进展，推动了自然语言转代码及代码摘要任务的应用。然而，模型容易产生与原意偏离的幻觉输出。代码摘要中的幻觉检测尤为困难，因编程语言与自然语言交互复杂。我们首次构建了包含约10,000个样本的专用数据集CodeSumEval，用于幻觉检测。提出实体追踪框架ETF：首先利用静态程序分析提取代码实体，再通过大模型映射并验证这些实体及其意图在生成摘要中的体现。实验表明该框架有效，达到73%的F1分数。该方法通过将摘要中的实体回溯至代码，可评估摘要准确性并精确定位错误位置。
原文摘要 · Abstract (English)
Recent advancements in large language models (LLMs) have significantly enhanced their ability to understand both natural language and code, driving their use in tasks like natural language-to-code (NL2Code) and code summarisation. However, LLMs are prone to hallucination, outputs that stray from intended meanings. Detecting hallucinations in code summarisation is especially difficult due to the complex interplay between programming and natural languages. We introduce a first-of-its-kind dataset, CodeSumEval, with ~10K samples, curated specifically for hallucination detection in code summarisation. We further propose a novel Entity Tracing Framework (ETF) that a) utilises static program analysis to identify code entities from the program and b) uses LLMs to map and verify these entities and their intents within generated code summaries. Our experimental analysis demonstrates the framework's effectiveness, leading to a 73% F1 score. The proposed approach provides a method for detecting hallucinations by tracing entities from the summary to the code, allowing us to evaluate summary accuracy and localise the error within the summary.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。