arXiv:2503.22776cs.SEcs.AI2025-03被引 3

用提示学习让预训练模型学会代码结构,无需重新训练。

Post-Incorporating Code Structural Knowledge into Pretrained Models via ICL for Code Translation

  • 通过信息覆盖率选择示例,提升代码结构知识注入精度。
  • 提出基于抽象语法树的近似算法,在多项式时间内求解最优示例集。
  • 首次实现不依赖训练的代码结构知识后置注入,适合现有模型快速升级。

代码翻译需在不同编程语言间迁移代码库。尽管大语言模型在软件挖掘中取得进展,但处理源代码的语法结构仍是挑战。传统语法感知方法依赖复杂模型架构与损失函数,集成成本高。本文采用上下文学习(ICL),将任务示例直接融入输入上下文,以测试时方式向预训练大模型注入代码结构知识。从信息论视角重新审视示例选择,提出基于信息覆盖的列表式选择策略,优于传统相似性与多样性结合的方法。为量化信息覆盖,引入抽象语法树覆盖度(CAST)作为代理指标,并证明其最大化问题是标准子模最大化问题。据此设计贪心算法,理论上保证(1-1/e)近似解,且时间复杂度为多项式级。本方法是首个无需训练、模型无关的测试时代码结构知识注入方案。实验表明其显著提升模型性能,揭示两点关键发现:1)代码结构知识可在推理阶段有效注入,即使训练中被忽略;2)扩大模型规模或训练数据不会自然涌现代码结构知识,凸显显式建模语法结构的必要性。

原文摘要 · Abstract (English)

Code translation migrates codebases across programming languages. Recently, large language models (LLMs) have achieved significant advancements in software mining. However, handling the syntactic structure of source code remains a challenge. Classic syntax-aware methods depend on intricate model architectures and loss functions, rendering their integration into LLM training resource-intensive. This paper employs in-context learning (ICL), which directly integrates task exemplars into the input context, to post-incorporate code structural knowledge into pre-trained LLMs. We revisit exemplar selection in ICL from an information-theoretic perspective, proposing that list-wise selection based on information coverage is more precise and general objective than traditional methods based on combining similarity and diversity. To address the challenges of quantifying information coverage, we introduce a surrogate measure, Coverage of Abstract Syntax Tree (CAST). Furthermore, we formulate the NP-hard CAST maximization for exemplar selection and prove that it is a standard submodular maximization problem. Therefore, we propose a greedy algorithm for CAST submodular maximization, which theoretically guarantees a (1-1/e)-approximate solution in polynomial time complexity. Our method is the first training-free and model-agnostic approach to post-incorporate code structural knowledge into existing LLMs at test time. Experimental results show that our method significantly improves LLMs performance and reveals two meaningful insights: 1) Code structural knowledge can be effectively post-incorporated into pre-trained LLMs during inference, despite being overlooked during training; 2) Scaling up model size or training data does not lead to the emergence of code structural knowledge, underscoring the necessity of explicitly considering code syntactic structure.

代码生成提示学习语法结构大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。