用可解释的评分方法，智能分配大模型微调的低秩参数量。
ShapLoRA: Allocation of Low-rank Adaption on Large Language Models via Shapley Value Inspired Importance Estimation
- 基于博弈论思想设计可解释的权重分配指标，替代传统不可靠评估。
- 在多个任务上表现优于现有方法，参数量相近但效果更优。
- 适合需要高效、透明微调方案的研究者和工程实践者。
低秩适配（LoRA）是参数高效微调（PEFT）的代表性方法，对普及大型语言模型至关重要。传统LoRA采用均匀分配秩，近期研究发现合理分配模型各层的秩可提升性能。然而，已有方法依赖不可解释且不可靠的重要性评估，存在局限性。为此，我们提出ShapLoRA框架：受可解释性度量Shapley Value启发，结合敏感性分析与协作博弈中联盟思想，提出更可解释的重要程度度量——Shapley敏感度。同时优化流程：(a) 在独立验证集上计算Shapley敏感度；(b) 设计分配-重训练流程以保证公平对比。在多个挑战性任务上的实验表明，该方法在可比可调参数量下超越近期基线。
原文摘要 · Abstract (English)
Low-rank adaption (LoRA) is a representative method in the field of parameter-efficient fine-tuning (PEFT), and is key to Democratizating the modern large language models (LLMs). The vanilla LoRA is implemented with uniform ranks, and the recent literature have found that properly allocating ranks on the LLM backbones results in performance boosts. However, the previous rank allocation methods have limitations since they rely on inexplanable and unreliable importance measures for the LoRA ranks. To address the above issues, we propose the ShapLoRA framework. Inspired by the explanable attribution measure Shapley Value, we combine the sensitivity-based measures with the idea of coalitions in the collaborative games among LoRA ranks, and propose a more explainable importance measure called Shapley sensitivity. In addition, we optimize the workflow of the existing works by: (a) calculating Shapley sensitivity on a separate validation set; (b) Setting up the allocating-retraining procedures for fair comparisons. We have conducted experiments on various challenging tasks, and the experimental results demonstrate that our ShapLoRA method can outperform the recent baselines with comparable tunable parameters.\footnote{Codes and fine-tuned models will be open-sourced to facilitate future research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。