动态筛选低秩适配中的有效子解，提升微调精度
BeamLoRA: Beam-Constraint Low-Rank Adaptation
- 将LoRA各秩视为候选解，搜索最优组合
- 固定秩下性能超越现有基线方法
- 适合需要高效高精度微调的场景
为满足大语言模型高效微调的需求，低秩适配（LoRA）已成为最有效的参数高效微调方法之一。然而，尽管提升了效率，其准确性仍有提升空间。本文从新视角分析LoRA秩的特性，发现不同秩在微调过程中重要性各异且动态变化，可能限制性能。为此提出BeamLoRA，将每个LoRA模块视为一个束，每秩对应一个潜在子解，微调过程即寻找最优子解组合。该方法动态淘汰表现差的子解，扩展有前景子解的参数空间，在固定秩下提升性能。在三个基础模型和12个数据集（涵盖数学推理、代码生成、常识推理）上的实验表明，BeamLoRA持续优于其他基线方法。
原文摘要 · Abstract (English)
Due to the demand for efficient fine-tuning of large language models, Low-Rank Adaptation (LoRA) has been widely adopted as one of the most effective parameter-efficient fine-tuning methods. Nevertheless, while LoRA improves efficiency, there remains room for improvement in accuracy. Herein, we adopt a novel perspective to assess the characteristics of LoRA ranks. The results reveal that different ranks within the LoRA modules not only exhibit varying levels of importance but also evolve dynamically throughout the fine-tuning process, which may limit the performance of LoRA. Based on these findings, we propose BeamLoRA, which conceptualizes each LoRA module as a beam where each rank naturally corresponds to a potential sub-solution, and the fine-tuning process becomes a search for the optimal sub-solution combination. BeamLoRA dynamically eliminates underperforming sub-solutions while expanding the parameter space for promising ones, enhancing performance with a fixed rank. Extensive experiments across three base models and 12 datasets spanning math reasoning, code generation, and commonsense reasoning demonstrate that BeamLoRA consistently enhances the performance of LoRA, surpassing the other baseline methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。