测试大模型在代码适配中识别上下文错误的能力，发现其表现不佳。
Coding in a Bubble? Evaluating LLMs in Resolving Context Adaptation Bugs During Code Adaptation
- 用大模型生成有上下文缺陷的代码来构建评估基准。
- 最佳模型仅55.93%通过率，30%性能下降，且常复制错误。
- 揭示大模型跨上下文推理能力不足，适合关注代码自动化研究者。
代码适配是软件开发中的基础但挑战性任务，需将现有代码修改以适应新环境。核心难题在于解决上下文适配错误（CtxBugs），即代码在原环境中正确，但在目标环境中违反约束。这类错误无法通过局部修复解决，需跨上下文推理识别语义不一致。忽略它们可能导致适配失败。尽管大语言模型（LLMs）在代码任务中展现潜力，但其处理CtxBugs的能力仍不明确。为此，本文提出CtxBugGen框架，利用大模型在缺乏上下文时生成看似合理但无上下文约束的代码特性，通过四步流程生成有效且相关的CtxBugs：(1) 选择适配任务，(2) 任务特定扰动，(3) 基于大模型的变体生成，(4) CtxBugs识别。基于该基准，我们对四个先进大模型进行了实证研究。结果表明其在化解CtxBugs方面表现不佳：最佳模型Kimi-K2在Pass@1上仅达55.93%，仅解决52.47%的错误；存在CtxBugs使大模型适配性能最高下降30%。故障分析显示，大模型常忽略错误并在输出中重复。本研究揭示了大模型在跨上下文推理上的关键缺陷，强调需提升其上下文感知能力以实现可靠的代码适配。
原文摘要 · Abstract (English)
Code adaptation is a fundamental but challenging task in software development, requiring developers to modify existing code for new contexts. A key challenge is to resolve Context Adaptation Bugs (CtxBugs), which occurs when code correct in its original context violates constraints in the target environment. Unlike isolated bugs, CtxBugs cannot be resolved through local fixes and require cross-context reasoning to identify semantic mismatches. Overlooking them may lead to critical failures in adaptation. Although Large Language Models (LLMs) show great potential in automating code-related tasks, their ability to resolve CtxBugs remains a significant and unexplored obstacle to their practical use in code adaptation. To bridge this gap, we propose CtxBugGen, a novel framework for generating CtxBugs to evaluate LLMs. Its core idea is to leverage LLMs' tendency to generate plausible but context-free code when contextual constraints are absent. The framework generates CtxBugs through a four-step process to ensure their relevance and validity: (1) Adaptation Task Selection, (2) Task-specific Perturbation,(3) LLM-based Variant Generation and (4) CtxBugs Identification. Based on the benchmark constructed by CtxBugGen, we conduct an empirical study with four state-of-the-art LLMs. Our results reveal their unsatisfactory performance in CtxBug resolution. The best performing LLM, Kimi-K2, achieves 55.93% on Pass@1 and resolves just 52.47% of CtxBugs. The presence of CtxBugs degrades LLMs' adaptation performance by up to 30%. Failure analysis indicates that LLMs often overlook CtxBugs and replicate them in their outputs. Our study highlights a critical weakness in LLMs' cross-context reasoning and emphasize the need for new methods to enhance their context awareness for reliable code adaptation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。