为LoRA设计自适应秩分配，压缩参数量75%-90%且不损失性能
Post-Optimization Adaptive Rank Allocation for LoRA
- 通过奇异值分解全局阈值剪枝，按层谱重要性非均匀分配秩
- 在多视觉与语言任务上压缩75%-90%参数，性能几乎不变
- 无需训练修改，可无缝集成到现有微调流程中
现代基础模型规模的指数级增长推动了低秩适配（LoRA）作为高效微调技术的广泛应用。然而，标准LoRA忽略各层固有维度差异，强制统一秩，导致参数冗余。我们提出后优化自适应秩分配（PARA），一种无需数据的LoRA压缩方法，可无缝集成至现有微调流程。PARA利用奇异值分解，基于跨所有层的全局奇异值阈值对LoRA秩进行剪枝，实现基于层间谱重要性的非均匀秩分配。作为后处理方法，PARA避免了动态架构带来的训练修改与稳定性问题。实验表明，PARA在多个视觉与语言基准上将参数量减少75%-90%，同时保持原始未压缩LoRA的预测性能。代码将在接受后发布。
原文摘要 · Abstract (English)
Exponential growth in the scale of modern foundation models has led to the widespread adoption of Low-Rank Adaptation (LoRA) as a parameter-efficient fine-tuning technique. However, standard LoRA implementations disregard the varying intrinsic dimensionality of model layers and enforce a uniform rank, leading to parameter redundancy. We propose Post-Optimization Adaptive Rank Allocation (PARA), a data-free compression method for LoRA that integrates seamlessly into existing fine-tuning pipelines. PARA leverages Singular Value Decomposition to prune LoRA ranks using a global threshold over singular values across all layers. This results in non-uniform rank allocation based on layer-wise spectral importance. As a post-hoc method, PARA circumvents the training modifications and resulting instabilities that dynamic architectures typically incur. We empirically demonstrate that PARA reduces parameter count by 75-90\% while preserving the predictive performance of the original, uncompressed LoRA across multiple vision and language benchmarks. Code will be published upon acceptance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。