无需参考即可检测代码审查中生成内容的幻觉，提升AI审查可信度。
HalluJudge: A Reference-Free Hallucination Detection for Context Misalignment in Code Review Automation
- 基于上下文一致性设计多分支推理框架，实现无参考幻觉检测。
- 在真实项目中达成0.85的F1分数，平均成本仅0.009美元。
- 67%的判断结果符合开发者实际偏好，适合企业级代码审查场景。
大型语言模型（LLMs）在代码审查自动化（如评论生成）中表现强劲，但存在生成内容脱离实际代码的幻觉问题，严重阻碍其在开发流程中的应用。为解决此问题，我们探索了无需参考的幻觉检测方法。本文提出HalluJudge，通过上下文对齐评估生成评论的可靠性，包含从直接判断到结构化多分支推理（如Tree-of-Thoughts）等四种策略。我们在Atlassian的企业级软件项目上进行全面评估，分析其有效性与成本效率。此外，我们对比了HalluJudge判断与真实生产环境中开发者偏好的一致性。结果表明，HalluJudge的幻觉检测在成本上具有优势，F1得分为0.85，平均成本为$0.009；平均67%的评估结果与开发者偏好一致。研究说明，HalluJudge可作为实用防护机制，减少开发者接触幻觉评论的风险，增强对AI辅助代码审查的信任。
原文摘要 · Abstract (English)
Large Language models (LLMs) have shown strong capabilities in code review automation, such as review comment generation, yet they suffer from hallucinations -- where the generated review comments are ungrounded in the actual code -- poses a significant challenge to the adoption of LLMs in code review workflows. To address this, we explore effective and scalable methods for a hallucination detection in LLM-generated code review comments without the reference. In this work, we design HalluJudge that aims to assess the grounding of generated review comments based on the context alignment. HalluJudge includes four key strategies ranging from direct assessment to structured multi-branch reasoning (e.g., Tree-of-Thoughts). We conduct a comprehensive evaluation of these assessment strategies across Atlassian's enterprise-scale software projects to examine the effectiveness and cost-efficiency of HalluJudge. Furthermore, we analyze the alignment between HalluJudge's judgment and developer preference of the actual LLM-generated code review comments in the real-world production. Our results show that the hallucination assessment in HalluJudge is cost-effective with an F1 score of 0.85 and an average cost of $0.009. On average, 67% of the HalluJudge assessments are aligned with the developer preference of the actual LLM-generated review comments in the online production. Our results suggest that HalluJudge can serve as a practical safeguard to reduce developers' exposure to hallucinated comments, fostering trust in AI-assisted code reviews.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。