arXiv:2507.10641cs.SEcs.AI2025-07

用大规模数据微调模型,提升对代码语义的理解能力。

A Code Comprehension Benchmark for Large Language Models for Code

  • 通过下游任务微调增强模型对代码语义的深层理解。
  • QWQ-32B模型在主观性评分任务中准确率从70%提升至83.47%。
  • Codestral-22B经DPO微调后达到87.66%最高准确率,适合语义分析场景。

大型语言模型在代码生成与补全等任务中表现优异,因其在大量代码数据上训练,且预训练目标为下一个词预测,倾向于学习代码的表面语法模式。然而,这并不保证具备代码理解能力——即捕捉代码语义的能力。我们认为,这是这些模型在需深层语义理解的任务(如调试、优化)中表现不佳的原因。为此,我们提出使用大规模数据集对模型进行特定于代码理解任务的微调,以增强其对代码语义的稳健理解。我们在一系列旨在评估超越表面语法匹配的代码理解任务上,评估了三种不同规模的代码模型。特别地,在主观性评分任务中,模型性能在微调后显著提升。其中,QWQ-32B模型准确率从70%提高到83.47%。其他模型也呈现类似或可解释的提升趋势,明确表明代码理解能力增强。在所研究模型中,经DPO微调的Codestral-22B在主观性评分任务上达到最高微平均准确率87.66%。

原文摘要 · Abstract (English)

Large Language Models have shown impressive capabilities in coding tasks like code generation and code completion, as they have been trained on a large amount of code data. Also, since one of the core pretraining objectives is Next Token Prediction, these models tends to learn surface-level syntactic patterns in code. However, this does not guarantee code comprehension ability i.e. the ability to capture the semantics of the code. In our opinion, this is the reason why these models often underperform on tasks that require deeper semantic understanding, such as code debugging and code optimization. To address this, we propose fine-tuning these models specifically for code comprehension tasks using large-scale datasets, enabling them to develop a more robust understanding of code semantics. We evaluate three code models of varying sizes on a suite of code comprehension tasks designed to assess semantic understanding beyond surface-level syntactic pattern matching. In particular, we analyze performance on the Subjectivity Grading Task and observe that model performance improves after fine-tuning on relevant downstream tasks. The most significant improvement is seen in the QWQ-32B model, where accuracy increases from 70% to 83.47%. A similar or explainable trend is observed across other models, clearly indicating an enhancement in code comprehension ability. Among the models studied, the DPO-fine-tuned Codestral-22B achieves the highest micro-accuracy of 87.66% on the Subjectivity Grading Task.

代码理解微调语义分析

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。