用程序运行轨迹增强大模型修复代码，提升修复效果
Towards Effectively Leveraging Execution Traces for Program Repair with Code LLMs
- 将程序运行时轨迹融入提示词，引导大模型理解动态行为
- 在部分配置下修复准确率提升，但复杂轨迹会削弱效果
- 适合关注代码修复中运行时信息利用的研究者
大型语言模型（LLMs）在自动程序修复（APR）等编程任务中表现优异，但现有方法多依赖静态分析，忽视程序的运行时行为。受知识增强自然语言处理启发，本文尝试通过在标准修复提示中加入程序执行轨迹来弥补这一缺陷。我们在三个主流APR数据集上使用GPT系列模型评估该方法。结果表明，仅简单将轨迹加入提示，仅在6组测试配置中的2组实现性能提升；且随着轨迹复杂度增加，其有效性显著下降。我们探索了多种轨迹提示策略，发现经大模型优化的提示能更一致地超越无轨迹基线。此外，基于轨迹的提示优于在小规模数据集上微调小型模型；探针实验也验证了执行轨迹可补充大模型的推理能力。
原文摘要 · Abstract (English)
Large Language Models (LLMs) show promising performance on various programming tasks, including Automatic Program Repair (APR). However, most approaches to LLM-based APR are limited to the static analysis of the programs, while disregarding their runtime behavior. Inspired by knowledge-augmented NLP, in this work, we aim to remedy this potential blind spot by augmenting standard APR prompts with program execution traces. We evaluate our approach using the GPT family of models on three popular APR datasets. Our findings suggest that simply incorporating execution traces into the prompt provides a limited performance improvement over trace-free baselines, in only 2 out of 6 tested dataset / model configurations. We further find that the effectiveness of execution traces for APR diminishes as their complexity increases. We explore several strategies for leveraging traces in prompts and demonstrate that LLM-optimized prompts help outperform trace-free prompts more consistently. Additionally, we show trace-based prompting to be superior to finetuning a smaller LLM on a small-scale dataset; and conduct probing studies reinforcing the notion that execution traces can complement the reasoning abilities of the LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。