改进低秩微调的缩放因子，让模型性能随秩增大而提升。
RoRA: Efficient Fine-Tuning of LLM with Reliability Optimization for Rank Adaptation
- 用α/√r替代α/r，优化低秩适配的缩放机制。
- 在未压缩和剪枝模型上均显著提升准确率，最高超LoRA 6.5%。
- 特别适合剪枝后模型的微调，性能优势明显。
微调有助于大语言模型恢复退化信息并提升任务表现。尽管低秩适配（LoRA）广泛应用且有效，但我们发现其缩放因子会限制甚至降低性能，尤其当秩增大时。为此，我们提出RoRA（Rank-adaptive Reliability Optimization），通过将$α/r$替换为$α/\sqrt{r}$，确保随着秩增大性能持续提升。RoRA不仅提升未压缩模型的微调效果，在剪枝模型的精度恢复这一更挑战的任务中也表现优异。大量实验表明，RoRA在LLaMA-7B/13B、LLaMA2-7B和LLaMA3-8B上平均准确率与鲁棒性均超越当前最优（SOTA），分别优于LoRA和DoRA 6.5%和2.9%。在剪枝模型微调中优势显著：对剪除81.4%参数的SHEARED-LLAMA-1.3（基于LLaMA-7B），RoRA平均准确率比LoRA高5.7%，比DoRA高3.9%。
原文摘要 · Abstract (English)
Fine-tuning helps large language models (LLM) recover degraded information and enhance task performance. Although Low-Rank Adaptation (LoRA) is widely used and effective for fine-tuning, we have observed that its scaling factor can limit or even reduce performance as the rank size increases. To address this issue, we propose RoRA (Rank-adaptive Reliability Optimization), a simple yet effective method for optimizing LoRA's scaling factor. By replacing $α/r$ with $α/\sqrt{r}$, RoRA ensures improved performance as rank size increases. Moreover, RoRA enhances low-rank adaptation in fine-tuning uncompressed models and excels in the more challenging task of accuracy recovery when fine-tuning pruned models. Extensive experiments demonstrate the effectiveness of RoRA in fine-tuning both uncompressed and pruned models. RoRA surpasses the state-of-the-art (SOTA) in average accuracy and robustness on LLaMA-7B/13B, LLaMA2-7B, and LLaMA3-8B, specifically outperforming LoRA and DoRA by 6.5% and 2.9% on LLaMA-7B, respectively. In pruned model fine-tuning, RoRA shows significant advantages; for SHEARED-LLAMA-1.3, a LLaMA-7B with 81.4% pruning, RoRA achieves 5.7% higher average accuracy than LoRA and 3.9% higher than DoRA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。