LLM在代码故障定位中对输入顺序敏感,顺序不当会导致准确率暴跌。
Order Matters! An Empirical Study on Large Language Models' Input Order Bias in Software Fault Localization
- 测试不同代码顺序对模型性能的影响,发现顺序差可致准确率下降超80%。
- 拆分输入上下文可将准确率差距从22%降至1%,显著缓解顺序偏差。
- 基于依赖图排序优于传统方法,适合提升代码故障定位效果。
大型语言模型(LLMs)在软件工程任务如故障定位(FL)和自动程序修复(APR)中展现出巨大潜力。本研究探讨输入顺序与上下文大小对LLM在FL中表现的影响。在包含Java和Python项目的两个基准上,采用肯德尔τ距离测试不同方法顺序,包括“理想”(真实答案靠前)和“最差”(真实答案靠后)情况。结果表明存在显著顺序偏倚:在Java项目中Top-1准确率从57%降至20%,在Python项目中从38%降至约3%。将输入拆分为更小上下文可减轻该偏倚,使性能差距从22%缩小至6%,最终仅剩1%。通过重命名方法名验证是否由数据泄露导致,结果仍保持一致趋势,说明偏倚非数据泄露所致。此外,基于依赖图(DepGraph)的排序达到48%的Top-1准确率,优于调用图深度优先搜索(CallGraphDFS)等简单方法。研究强调输入结构、上下文管理与排序策略对提升LLM在软件工程任务中性能的重要性。
原文摘要 · Abstract (English)
Large Language Models (LLMs) show great promise in software engineering tasks like Fault Localization (FL) and Automatic Program Repair (APR). This study investigates the impact of input order and context size on LLM performance in FL, a crucial step for many downstream software engineering tasks. We test different orders for methods using Kendall Tau distances, including "perfect" (where ground truths come first) and "worst" (where ground truths come last), using two benchmarks that consist of both Java and Python projects. Our results indicate a significant bias in order; Top-1 FL accuracy in Java projects drops from 57% to 20%, while in Python projects, it decreases from 38% to approximately 3% when we reverse the code order. Breaking down inputs into smaller contexts helps reduce this bias, narrowing the performance gap in FL from 22% to 6% and then to just 1% on both benchmarks. We then investigated whether the bias in order was caused by data leakage by renaming the method names with more meaningful alternatives. Our findings indicated that the trend remained consistent, suggesting that the bias was not due to data leakage. We also look at ordering methods based on traditional FL techniques and metrics. Ordering using DepGraph's ranking achieves 48% Top-1 accuracy, which is better than more straightforward ordering approaches like CallGraphDFS. These findings underscore the importance of how we structure inputs, manage contexts, and choose ordering methods to improve LLM performance in FL and other software engineering tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。