ABBA通过双低秩矩阵重构提升大模型微调表达能力
ABBA-Adapters: Efficient and Expressive Fine-Tuning of Foundation Models
- 用两个独立可训练的低秩矩阵做哈达玛积，实现更新与预训练权重解耦
- 在相同参数量下，重建精度和推理任务表现均超越现有方法
- 适合需要高效高表达力微调的科研与工业场景
大语言模型在多种任务中表现优异，但高效适配新领域仍是关键挑战。参数高效微调（PEFT）通过引入轻量级可训练模块，在保持大部分预训练权重固定的同时实现适应。主流方法LoRA采用低秩分解建模更新，但其表达能力受限于秩的大小。近期方法如HiRA通过与冻结权重的哈达玛积提升表达力，但仍依赖预训练模型结构。本文提出ABBA，将更新重参数化为两个独立可学习的低秩矩阵的哈达玛积。与先前工作不同，ABBA完全解耦更新与预训练权重，使两部分可自由优化。该设计在相同参数预算下显著提升表达力，经矩阵重构实验验证。实证表明，ABBA在算术与常识推理基准上达到当前最优性能，跨多个模型一致显著优于现有PEFT方法。代码已公开于：https://github.com/CERT-Lab/abba。
原文摘要 · Abstract (English)
Large Language Models have demonstrated strong performance across a wide range of tasks, but adapting them efficiently to new domains remains a key challenge. Parameter-Efficient Fine-Tuning (PEFT) methods address this by introducing lightweight, trainable modules while keeping most pre-trained weights fixed. The prevailing approach, LoRA, models updates using a low-rank decomposition, but its expressivity is inherently constrained by the rank. Recent methods like HiRA aim to increase expressivity by incorporating a Hadamard product with the frozen weights, but still rely on the structure of the pre-trained model. We introduce ABBA, a new PEFT architecture that reparameterizes the update as a Hadamard product of two independently learnable low-rank matrices. In contrast to prior work, ABBA fully decouples the update from the pre-trained weights, enabling both components to be optimized freely. This leads to significantly higher expressivity under the same parameter budget, a property we validate through matrix reconstruction experiments. Empirically, ABBA achieves state-of-the-art results on arithmetic and commonsense reasoning benchmarks, consistently outperforming existing PEFT methods by a significant margin across multiple models. Our code is publicly available at: https://github.com/CERT-Lab/abba.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。