揭示大模型在长代码理解中依赖模式匹配而非语义理解的弱点
Sense and Sensitivity: Examining the Influence of Semantic Recall on Long Context Code Understanding
- 区分字面回忆与语义回忆，提出语义敏感度评估新方法
- 发现模型在长代码中间位置时语义召回率下降92.73%，远超现有基准
- 设计新任务SemTrace，有效暴露模型对语义理解的依赖缺陷
大型语言模型（LLMs）在理解大型代码库中的应用日益广泛，但其是否真正理解长代码上下文的操作语义，还是仅依赖模式匹配的捷径尚不明确。本文区分了字面回忆（原样检索代码）和语义回忆（理解操作语义）。评估10个前沿大模型发现，尽管先进模型在位置无关的字面回忆上表现接近完美，但在长上下文中居中位置时，语义回忆严重退化。为此，我们引入语义回忆敏感度，用以衡量任务是否需要真正理解代码操作语义，或允许模式匹配捷径。通过新颖的反事实测量方法，我们发现当前代码理解基准严重依赖模式匹配。我们提出新任务SemTrace，通过不可预测操作实现高语义敏感度；模型准确率在相关代码片段接近输入中心时出现严重位置效应，中位数下降达92.73%，显著高于CRUXEval的53.36%。研究结果表明，现有评估严重低估了长上下文代码理解中的语义回忆失败。
原文摘要 · Abstract (English)
Large language models (LLMs) are increasingly deployed for understanding large codebases, but whether they understand operational semantics of long code context or rely on pattern matching shortcuts remains unclear. We distinguish between lexical recall (retrieving code verbatim) and semantic recall (understanding operational semantics). Evaluating 10 state-of-the-art LLMs, we find that while frontier models achieve near-perfect, position-independent lexical recall, semantic recall degrades severely when code is centrally positioned in long contexts. We introduce semantic recall sensitivity to measure whether tasks require understanding of code's operational semantics vs. permit pattern matching shortcuts. Through a novel counterfactual measurement method, we show that models rely heavily on pattern matching shortcuts to solve existing code understanding benchmarks. We propose a new task SemTrace, which achieves high semantic recall sensitivity through unpredictable operations; LLMs' accuracy exhibits severe positional effects, with median accuracy drops of 92.73% versus CRUXEval's 53.36% as the relevant code snippet approaches the middle of the input code context. Our findings suggest current evaluations substantially underestimate semantic recall failures in long context code understanding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。