分析静态检测对代码库幻觉的识别能力，发现其可查出部分问题但仍有明显局限。
An Empirical Analysis of Static Analysis Methods for Detection and Mitigation of Code Library Hallucinations
- 用静态分析工具扫描生成代码，检测其中虚构的库函数调用
- 可识别16%-70%的错误，对库幻觉识别率达14%-85%
- 适合关注低成本检测方案的研究者与开发者使用
尽管研究众多，大型语言模型在生成代码时仍频繁出现幻觉，尤其是在使用代码库时。在需要调用库的自然语言转代码基准测试中，我们发现LLM生成的代码会在8.1%-40%的情况下使用不存在的库功能。一种直观的检测与缓解方法是采用静态分析。本文分析了静态分析工具在解决和无法解决此类问题方面的潜力。结果表明，静态分析工具可检测16%-70%的各类错误，以及14%-85%的库幻觉，具体表现随LLM和数据集不同而异。通过人工分析，我们识别出静态方法无法捕捉的案例，这为它们的上限性能设定了48.5%至77%的范围。总体而言，静态分析是一种成本低廉的应对部分幻觉的方法，但也量化了其始终无法彻底解决问题的局限性。
原文摘要 · Abstract (English)
Despite extensive research, Large Language Models continue to hallucinate when generating code, particularly when using libraries. On NL-to-code benchmarks that require library use, we find that LLMs generate code that uses non-existent library features in 8.1-40% of responses. One intuitive approach for detection and mitigation of hallucinations is static analysis. In this paper, we analyse the potential of static analysis tools, both in terms of what they can solve and what they cannot. We find that static analysis tools can detect 16-70% of all errors, and 14-85% of library hallucinations, with performance varying by LLM and dataset. Through manual analysis, we identify cases a static method could not plausibly catch, which gives an upper bound on their potential from 48.5% to 77%. Overall, we show that static analysis methods are cheap method for addressing some forms of hallucination, and we quantify how far short of solving the problem they will always be.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。