arXiv:2509.11686cs.SEcs.AI2025-09EMNLP被引 3

探究执行轨迹对代码大模型推理能力的作用,发现其实际帮助有限。

Do Code Semantics Help? A Comprehensive Study on Execution Trace-Based Information for Code Large Language Models

  • 构建通用框架,将执行轨迹等语义信息融入代码任务提示中。
  • 实验表明,语义信息对微调和推理阶段提升效果不明显。
  • 挑战了以往认为语义信息能显著增强模型能力的普遍认知。

代码大语言模型(Code LLM)在编程领域展现出强大能力,但其在推理程序运行行为和理解程序实际功能方面存在显著局限,制约了后续训练与实际应用。主要问题包括:(1) 难以准确分析程序运行时的实际行为;(2) 现有方法对执行轨迹等语义信息的表示不一致且碎片化,影响模型泛化与推理能力。为此,本文提出一个通用框架,支持将执行轨迹等语义信息融入代码任务相关提示,并系统研究其在提升Code LLM推理能力中的作用。重点考察基于轨迹的语义信息在监督微调(SFT)和推理阶段的效果。实验结果出人意料,与先前研究相反,表明语义信息对SFT及测试时扩展的帮助极为有限。

原文摘要 · Abstract (English)

Code Large Language Models (Code LLMs) have opened a new era in programming with their impressive capabilities. However, recent research has revealed critical limitations in their ability to reason about runtime behavior and understand the actual functionality of programs, which poses significant challenges for their post-training and practical deployment. Specifically, Code LLMs encounter two principal issues: (1) a lack of proficiency in reasoning about program execution behavior, as they struggle to interpret what programs actually do during runtime, and (2) the inconsistent and fragmented representation of semantic information, such as execution traces, across existing methods, which hinders their ability to generalize and reason effectively. These challenges underscore the necessity for more systematic approaches to enhance the reasoning capabilities of Code LLMs. To address these issues, we introduce a generic framework to support integrating semantic information~(e.g., execution trace) to code task-relevant prompts, and conduct a comprehensive study to explore the role of semantic information in enhancing the reasoning ability of Code LLMs accordingly. Specifically, we focus on investigating the usefulness of trace-based semantic information in boosting supervised fine-tuning~(SFT) and post-phase inference of Code LLMs. The experimental results surprisingly disagree with previous works and demonstrate that semantic information has limited usefulness for SFT and test time scaling of Code LLM.

代码生成大模型推理能力

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。