用语义保持的代码变换构建生成式表示,提升代码优化效果
SemRep: Generative Code Representation Learning with Code Transformations
- 以语义不变的代码变换作为中间表示,指导模型学习
- 正确率提升6.9%,泛化能力提高13.9%,推理计算减少25%
- 适合做进化式代码搜索,可发现大模型遗漏的优化
代码变换是软件开发中的基础能力,其效果依赖于高质量的代码表示以刻画语义并引导变换。现有方法将代码变换视为端到端学习任务,使表示构建隐含于模型权重中或依赖僵化的编译器级抽象。我们提出SemRep框架,通过生成式代码表示学习改进代码变换。核心思路是将语义保持的变换作为中间表示,既作为训练中的生成任务,又为后续指令特定变换提供指导。在通用代码编辑与优化任务(如GPU核优化)中,SemRep在相同训练预算下优于广泛微调的基线模型:正确率提升6.9%,性能提升1.1倍,泛化能力提高13.9%,鲁棒性提升6.7%。通过更充分探索多样代码变换,SemRep特别适用于进化搜索。结合进化编码代理,其发现的优化比685B参数的大模型未察觉,且达到相同性能时推理计算减少25%。
原文摘要 · Abstract (English)
Code transformation is a foundational capability in the software development process, where its effectiveness relies on constructing a high-quality code representation to characterize the input code semantics and guide the transformation. Existing approaches treat code transformation as an end-to-end learning task, leaving the construction of the representation needed for semantic reasoning implicit in model weights or relying on rigid compiler-level abstractions. We present SemRep, a framework that improves code transformation through generative code representation learning. Our key insight is to employ the semantics-preserving transformations as the intermediate representation, which serves as both a generative mid-training task and the guidance for subsequent instruction-specific code transformations. Across general code editing and optimization tasks (e.g., GPU kernel optimization), SemRep outperforms the extensively finetuned baselines with strictly the same training budget by 6.9% in correctness, 1.1x in performance, 13.9% in generalization, and 6.7% in robustness. With the improved exploration of diverse code transformations, SemRep is particularly amenable to evolutionary search. Combined with an evolutionary coding agent, SemRep finds optimizations that 685B larger-weight baselines fail to discover while achieving the same performance with 25% less inference compute.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。