解释性AI能显著影响开发者对代码审查AI的信任与认同。
Evaluating the Impact of Explainable AI on Trust in AI-Assisted Code Review
- 对比三种解释程度的AI审查系统，测试开发者信任与同意度变化。
- 详细解释提升信任（4.0/5），但适度解释更促共识（89.2%同意）。
- 解释越多越引发质疑，适合关注可解释性的开发团队使用。
背景：大语言模型（LLMs）正被用于自动化代码审查，但其决策依据难以理解。开发者难以评估生成建议的有效性，从而难以判断信任程度。解释性AI（XAI）在代码审查中的作用及其对信任的影响尚不明确。目标：研究XAI对开发者信任的影响。方法：开展包含34名参与者的组内用户研究，比较三种不同解释支持水平的基于LLM的代码审查系统：条件A（详细解释与反馈）、条件B（仅反馈）、条件C（无解释）。参与者结合真实代码变更请求审阅AI生成的审查意见。测量信任感知、对AI推荐的同意度、决策理由及耗时。结果：解释程度显著影响信任与同意度，但方式不同。完整解释（A）带来最高信任感知（均值3.99/5），但非最高同意率；适度解释（B）达成最高同意率（89.22%），可能因更多解释促使开发者质疑建议。无解释（C）导致最低信任与同意。解释水平未显著影响审查时间。最常见决策理由为代码可读性与正确性。结论：在代码审查中引入XAI会显著改变信任感知与对推荐的认同。该结果有助于设计可信的基于AI的代码审查系统，并指导人机协作软件开发中的认知研究。
原文摘要 · Abstract (English)
Background: Large language models (LLMs) are increasingly used to automate code review, but the reasoning behind their decisions remains hard to understand. Developers struggle to assess the validity of LLM-generated reviews, making it difficult to gauge how much trust to place in them. The role of Explainable AI (XAI) in code review and its impact on trust remain underexplored. Objective: We study the influence of XAI on developer trust in AI-assisted code reviews. Method: We conducted a within-subjects user study with 34 participants, comparing three LLM-based code review systems with varying levels of XAI support: Condition A (detailed explanation and review feedback), Condition B (review feedback only), and Condition C (no explanations). Participants reviewed real-world code change requests alongside the AI-generated reviews. We measured trust perceptions, agreement with the AI recommendation, the reasoning given for each decision, and the time taken. Results: The level of explanation significantly influences both trust and agreement with AI recommendations, but in different ways. Full explanations (A) yield the highest perceived trust (M = 3.99/5) but not the highest agreement, whereas moderate explanations (B) achieve the highest agreement (89.22%). This could suggest that more explanation prompts developers to question AI recommendations more frequently. No explanations (C) results in the lowest trust and agreement. Explanation level did not significantly affect review time. The most commonly cited reasons for decisions were code readability and correctness. Conclusion: Incorporating XAI into code review significantly changes trust perceptions and agreement with AI recommendations. These results inform the design and evaluation of trustworthy AI-based code review systems, as well as studies on the human factors of AI-assisted software development.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。