用语法树结构增强代码模型的层次位置表示
Seamlessly Integrating Tree-Based Positional Embeddings into Transformer Models for Source Code Representation
- 基于抽象语法树设计树形位置嵌入，编码节点深度与兄弟索引
- 在代码克隆检测任务中，准确率、F1等指标全面优于基线
- 适合需要理解代码深层结构的程序分析场景
基于Transformer的模型在代码表征任务中表现优异，但传统位置嵌入难以捕捉代码固有的层次结构，通常以抽象语法树（AST）形式表示。为此，我们提出一种新的树形位置嵌入方法，显式编码来自AST的层次关系，包括节点深度和兄弟索引。这些层次嵌入被整合进Transformer架构，特别用于增强CodeBERTa模型。通过掩码语言建模（MLM）预训练和代码克隆检测微调任务进行充分评估。实验结果表明，所提的树增强CodeBERTa在损失、准确率、F1分数、精确率和召回率各项指标上均持续优于基线模型，凸显了将显式结构信息融入基于Transformer的代码表征的重要性。
原文摘要 · Abstract (English)
Transformer-based models have demonstrated significant success in various source code representation tasks. Nonetheless, traditional positional embeddings employed by these models inadequately capture the hierarchical structure intrinsic to source code, typically represented as Abstract Syntax Trees (ASTs). To address this, we propose a novel tree-based positional embedding approach that explicitly encodes hierarchical relationships derived from ASTs, including node depth and sibling indices. These hierarchical embeddings are integrated into the transformer architecture, specifically enhancing the CodeBERTa model. We thoroughly evaluate our proposed model through masked language modeling (MLM) pretraining and clone detection fine-tuning tasks. Experimental results indicate that our Tree-Enhanced CodeBERTa consistently surpasses the baseline model in terms of loss, accuracy, F1 score, precision, and recall, emphasizing the importance of incorporating explicit structural information into transformer-based representations of source code.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。