根据权重敏感度动态分配低秩参数，提升大模型微调效率与稳定性。
Sensitivity-LoRA: Low-Load Sensitivity-Based Fine-Tuning for Large Language Models
- 基于损失函数的二阶导数动态分配低秩矩阵秩值
- 在多个任务上实现更高精度且计算开销极小
- 适合资源受限环境下的高效模型微调
大型语言模型（LLMs）已深刻改变日常生活与科研实践。然而，在资源受限环境下将通用模型适配到特定任务仍具挑战性。低秩适应（LoRA）作为参数高效微调（PEFT）的重要方法，通过低秩分解近似权重更新，但其对每个增量矩阵采用统一秩值，且现有秩分配方法计算复杂、不稳定，难以实用。为此，本文提出敏感度-LoRA，一种基于全局与局部敏感度动态分配秩值的高效微调方法。该方法利用损失函数的二阶导数（海塞矩阵）有效捕捉权重敏感度，实现最优秩分配且计算开销极小。实验表明，Sensitivity-LoRA在多种任务和基准测试中均展现出优异的性能、效率与稳定性。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have transformed both everyday life and scientific research. However, adapting LLMs from general-purpose models to specialized tasks remains challenging, particularly in resource-constrained environments. Low-Rank Adaptation (LoRA), a prominent method within Parameter-Efficient Fine-Tuning (PEFT), has emerged as a promising approach to LLMs by approximating model weight updates using low-rank decomposition. However, LoRA is limited by its uniform rank ( r ) allocation to each incremental matrix, and existing rank allocation techniques aimed at addressing this issue remain computationally inefficient, complex, and unstable, hindering practical applications. To address these limitations, we propose Sensitivity-LoRA, an efficient fine-tuning method that dynamically allocates ranks to weight matrices based on both their global and local sensitivities. It leverages the second-order derivatives (Hessian Matrix) of the loss function to effectively capture weight sensitivity, enabling optimal rank allocation with minimal computational overhead. Our experimental results have demonstrated robust effectiveness, efficiency and stability of Sensitivity-LoRA across diverse tasks and benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。