arXiv:2601.22905cs.LG2026-01被引 2

FlexLoRA动态调整低秩适配秩,更灵活高效。

FlexLoRA: Entropy-Guided Flexible Low-Rank Adaptation

  • 用谱能量熵评估矩阵重要性,实现细粒度秩分配
  • 支持全局预算下秩的动态剪枝与扩展,性能更优
  • 新增方向零影响初始化,提升训练稳定性

大预训练模型在多个领域表现卓越,但全量微调成本高昂。参数高效微调(PEFT)成为主流,其中低秩适配(LoRA)引入可训练低秩矩阵,表现优异,但固定秩设计缺乏灵活性。动态秩分配方法虽能缓解此问题,却依赖启发式元素级度量,全局排序秩方向且无机制扩展需额外适应的层。为此,我们提出熵引导的灵活低秩适配框架FlexLoRA:(i) 通过谱能量熵评估矩阵重要性;(ii) 支持在全局预算下进行秩剪枝与扩展;(iii) 采用零影响初始化新添加的奇异方向以保证稳定性。通过解决粒度、灵活性与稳定性问题,FlexLoRA提供了更系统的PEFT方案。大量实验表明,FlexLoRA在多个基准上持续优于现有最优基线。代码已开源:https://github.com/Chongjie-Si/Subspace-Tuning。

原文摘要 · Abstract (English)

Large pre-trained models achieve remarkable success across diverse domains, yet fully fine-tuning incurs prohibitive computational and memory costs. Parameter-efficient fine-tuning (PEFT) has thus become a mainstream paradigm. Among them, Low-Rank Adaptation (LoRA) introduces trainable low-rank matrices and shows strong performance, nevertheless, its fixed-rank design limits flexibility. Dynamic rank allocation methods mitigate this issue by pruning redundant directions; however, they often rely on heuristic, element-level metrics that globally sort rank directions without matrix-wise distinction, and they lack mechanisms to expand capacity in layers requiring additional adaptation. To overcome these limitations, we propose FlexLoRA, an entropy-guided flexible low-rank adaptation framework that (i) evaluates matrix importance via spectral energy entropy, (ii) supports rank pruning and expansion under a global budget, and (iii) employs zero-impact initialization for newly added singular directions to ensure stability. By addressing granularity, flexibility, and stability limitations, FlexLoRA provides a more principled solution for PEFT. Extensive experiments show that FlexLoRA consistently outperforms state-of-the-art baselines across benchmarks. Codes are available at https://github.com/Chongjie-Si/Subspace-Tuning.

参数高效低秩适配动态调整

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。