arXiv:2412.13467cs.SEcs.AI2024-12被引 1

用代码图结构高效适配大模型,节省99%内存

Transducer Tuning: Efficient Model Adaptation for Software Tasks Using Code Property Graphs

  • 通过代码属性图增强代码嵌入,仅优化轻量模块
  • 三任务测试中性能媲美全参数微调,参数量减少99%
  • 适合资源受限场景,比LoRA等方法更省参数

大型语言模型在多种软件工程任务中表现出色。然而,随着模型规模增大,微调带来的可训练参数增长使资源受限环境下的应用变得困难。本文提出一种基于代码属性图(CPGs)的模型适配技术——Transducer Tuning。该方法引入模块化组件“Transducer”,包含图向量化引擎(GVE)和注意力融合层(ABFL),将代码的结构与依赖信息编码为图特征向量,并融合至大模型的初始代码嵌入中。通过仅优化Transducer模块,无需对整个模型进行微调即可提升下游任务表现。我们在代码摘要生成、断言生成和代码翻译三个任务上验证了该方法,结果表明其性能可媲美全参数微调,同时将可训练参数减少高达99%;相比LoRA、Prompt-Tuning、Prefix-Tuning等方法,仅需1.5%–80%的可训练参数,仍保持竞争力。研究显示,通过Transducer Tuning融合结构信息,能实现更高效的模型适配,显著降低资源消耗。

原文摘要 · Abstract (English)

Large language models have demonstrated promising performance across various software engineering tasks. While fine-tuning is a common practice to adapt these models for downstream tasks, it becomes challenging in resource-constrained environments due to increased memory requirements from growing trainable parameters in increasingly large language models. We introduce \approach, a technique to adapt large models for downstream code tasks using Code Property Graphs (CPGs). Our approach introduces a modular component called \transducer that enriches code embeddings with structural and dependency information from CPGs. The Transducer comprises two key components: Graph Vectorization Engine (GVE) and Attention-Based Fusion Layer (ABFL). GVE extracts CPGs from input source code and transforms them into graph feature vectors. ABFL then fuses those graphs feature vectors with initial code embeddings from a large language model. By optimizing these transducers for different downstream tasks, our approach enhances the models without the need to fine-tune them for specific tasks. We have evaluated \approach on three downstream tasks: code summarization, assert generation, and code translation. Our results demonstrate competitive performance compared to full parameter fine-tuning while reducing up to 99\% trainable parameters to save memory. \approach also remains competitive against other fine-tuning approaches (e.g., LoRA, Prompt-Tuning, Prefix-Tuning) while using only 1.5\%-80\% of their trainable parameters. Our findings show that integrating structural and dependency information through Transducer Tuning enables more efficient model adaptation, making it easier for users to adapt large models in resource-constrained settings.

模型适配代码图低资源轻量化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。