R-LoRA通过随机化多头提升大模型多任务学习效果，节省显存和训练时间。
R-LoRA: Randomized Multi-Head LoRA for Efficient Multi-Task Learning
- 引入多头随机化，通过丢弃与随机初始化增强头矩阵多样性。
- 在多任务场景下性能显著提升，同时降低30%以上显存占用。
- 适合需要高效多任务微调的工业级应用，如跨领域智能助手。
微调大语言模型（LLM）计算成本高昂，低秩适配（LoRA）通过低秩矩阵近似权重更新提供低成本解决方案。在真实场景中，LLM需在多个领域数据上微调以实现跨领域任务，即多任务学习（MTL），但传统LoRA在此类复杂场景中表现不佳。为此，我们提出R-LoRA，引入多头随机化机制，通过多头丢弃与多头随机初始化多样化头矩阵，更高效地学习任务特定特征，同时保持共享知识表示。实验表明，R-LoRA不仅提升多任务学习性能，还减少显存使用与训练时间。其增益源于头矩阵多样性的提升，验证了该方法的有效性。代码已开源：https://github.com/jinda-liu/R-LoRA。
原文摘要 · Abstract (English)
Fine-tuning large language models (LLMs) is computationally expensive, and Low-Rank Adaptation (LoRA) provides a cost-effective solution by approximating weight updates through low-rank matrices. In real-world scenarios, LLMs are fine-tuned on data from multiple domains to perform tasks across various fields, embodying multi-task learning (MTL). LoRA often underperforms in such complex scenarios. To enhance LoRA's capability in multi-task learning, we propose R-LoRA, which incorporates Multi-Head Randomization. Multi-Head Randomization diversifies the head matrices through Multi-Head Dropout and Multi-Head Random Initialization, enabling more efficient learning of task-specific features while maintaining shared knowledge representation. Our approach not only improves performance in MTL but also reduces GPU memory usage and training time. Experiments show that R-LoRA's gains stem from increased diversity in the head matrices, demonstrating its effectiveness for multi-task learning. The code is available at https://github.com/jinda-liu/R-LoRA
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。