用多秩混合提升低秩微调性能，参数更少、效果更好。
MoR: Mixture of Ranks for Low-Rank Adaptation Tuning
- 基于输入动态融合多个低秩组件，模拟高秩表达
- 参数仅需基线93.93%却提升1.31%性能
- 适合追求高效多任务微调的研究者
低秩微调（LoRA）旨在使性能接近全量微调，但仍面临两大挑战：(1) 单纯增大LoRA秩无法有效捕捉高秩信息，导致性能瓶颈；(2) 类MoE的LoRA方法显著增加参数与推理延迟，违背高效微调初衷。为此，本文提出多秩混合（MoR），根据输入学习不同任务的秩相关信息，并高效融合多秩特征。我们首次将多个LoRA的集成等价于扩展LoRA秩。此外，我们假设低秩LoRA已捕获足够内在信息，可通过数学变换推导出高秩信息，从而降低学习难度并增强多任务能力。实验表明，MoR在仅使用基线93.93%参数的情况下，实现1.31%的性能提升。
原文摘要 · Abstract (English)
Low-Rank Adaptation (LoRA) drives research to align its performance with full fine-tuning. However, significant challenges remain: (1) Simply increasing the rank size of LoRA does not effectively capture high-rank information, which leads to a performance bottleneck.(2) MoE-style LoRA methods substantially increase parameters and inference latency, contradicting the goals of efficient fine-tuning and ease of application. To address these challenges, we introduce Mixture of Ranks (MoR), which learns rank-specific information for different tasks based on input and efficiently integrates multi-rank information. We firstly propose a new framework that equates the integration of multiple LoRAs to expanding the rank of LoRA. Moreover, we hypothesize that low-rank LoRA already captures sufficient intrinsic information, and MoR can derive high-rank information through mathematical transformations of the low-rank components. Thus, MoR can reduces the learning difficulty of LoRA and enhances its multi-task capabilities. MoR achieves impressive results, with MoR delivering a 1.31\% performance improvement while using only 93.93\% of the parameters compared to baseline methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。