用图模型选关键参数，让大模型微调更省参且高效
Adaptive Principal Components Allocation with the $\ell_{2,g}$-regularized Gaussian Graphical Model for Efficient Fine-Tuning Large Models
- 基于高斯图模型构建参数重要性评估机制
- 在GLUE上用少得多的可训练参数达到竞争力表现
- 适合资源有限时高效微调大模型的研究者
本文提出一种基于高斯图模型（GGM）的新颖参数高效微调（PEFT）方法，据我们所知为首次将GGM应用于PEFT任务。该方法利用ℓ₂,𝑔-范数有效筛选关键参数并捕捉全局依赖关系，通过块坐标下降（BCD）算法高效求解由此产生的非凸优化问题。在微调RoBERTa-Base模型时，于GLUE基准测试上验证了其有效性，仅使用极少可训练参数即实现具有竞争力的性能。相关代码已公开：https://github.com/jzheng20/Course projects.git。
原文摘要 · Abstract (English)
In this work, we propose a novel Parameter-Efficient Fine-Tuning (PEFT) approach based on Gaussian Graphical Models (GGMs), marking the first application of GGMs to PEFT tasks, to the best of our knowledge. The proposed method utilizes the $\ell_{2,g}$-norm to effectively select critical parameters and capture global dependencies. The resulting non-convex optimization problem is efficiently solved using a Block Coordinate Descent (BCD) algorithm. Experimental results on the GLUE benchmark [24] for fine-tuning RoBERTa-Base [18] demonstrate the effectiveness of the proposed approach, achieving competitive performance with significantly fewer trainable parameters. The code for this work is available at: https://github.com/jzheng20/Course projects.git.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。