让大模型理解代码结构，提升编程理解能力
GALLa: Graph Aligned Large Language Models for Improved Source Code Understanding
- 用图神经网络注入代码结构信息，不修改原有模型架构
- 在5个任务上显著提升7个不同规模模型的性能
- 训练时需结构数据，推理时零开销，适合各类代码任务
编程语言蕴含丰富的语义信息（如数据流），以图的形式表达，而非仅通过源码表面文本体现。当前代码大模型虽已扩展至数十亿参数，但仅将代码视为文本标记，忽略结构信息；而部分建模结构信息的模型需修改Transformer架构，限制其规模与预训练大模型的兼容性。本文提出GALLa——图对齐大语言模型，利用图神经网络与跨模态对齐技术，在微调阶段将代码结构信息作为辅助任务注入现有代码LLM中。该框架具备模型无关与任务无关特性，可适配任意代码大模型及下游任务，只需在训练时使用无关语料的结构图数据，推理时无额外开销。在五个代码任务上，针对七个规模从350M到14B的基线模型进行实验，验证了GALLa的有效性，即使在LLaMA3和Qwen2.5-Coder等强模型上也实现持续提升。
原文摘要 · Abstract (English)
Programming languages possess rich semantic information - such as data flow - that is represented by graphs and not available from the surface form of source code. Recent code language models have scaled to billions of parameters, but model source code solely as text tokens while ignoring any other structural information. Conversely, models that do encode structural information of code make modifications to the Transformer architecture, limiting their scale and compatibility with pretrained LLMs. In this work, we take the best of both worlds with GALLa - Graph Aligned Large Language Models. GALLa utilizes graph neural networks and cross-modal alignment technologies to inject the structural information of code into LLMs as an auxiliary task during finetuning. This framework is both model-agnostic and task-agnostic, as it can be applied to any code LLM for any code downstream task, and requires the structural graph data only at training time from a corpus unrelated to the finetuning data, while incurring no cost at inference time over the baseline LLM. Experiments on five code tasks with seven different baseline LLMs ranging in size from 350M to 14B validate the effectiveness of GALLa, demonstrating consistent improvement over the baseline, even for powerful models such as LLaMA3 and Qwen2.5-Coder.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。