用轻量动态插件实现多LoRA高效融合，提升大模型多任务推理速度与精度。
DLP-LoRA: Efficient Task-Specific LoRA Fusion with a Dynamic, Lightweight Plugin for Large Language Models
- 设计500万参数小MLP，按句子级动态融合多个LoRA，避免逐令牌计算
- 26项任务测试平均准确率92.34%，问答任务BLEU/ROUGE显著提升
- 推理耗时不足单个LoRA的两倍，适合需快速切换任务的部署场景
大型语言模型（LLM）在多种任务上表现优异，但针对特定领域微调仍需大量资源。参数高效微调（PEFT）方法如低秩适应（LoRA）通过微调少量参数缓解此问题。然而，现有多个LoRA融合方法缺乏基于上下文的动态机制，且常因逐令牌操作导致推理延迟。我们提出DLP-LoRA，一种动态轻量插件，采用仅500万参数的微型MLP模块，结合top-p采样策略，在句子级别动态融合多个LoRA。该方法通过并行计算将推理时间控制在单个LoRA的两倍以内。在26个任务（包括多项选择题和问答）上的评估表明，DLP-LoRA在多项选择数据集上平均准确率达92.34%，在问答数据集上显著提升BLEU和ROUGE分数，优于不同LLM骨干网络在复合任务设置下的表现。DLP-LoRA有效平衡性能与效率，为LLM的动态多任务适配提供实用解决方案。代码已开源：https://github.com/MeCuping/DLP-LoRA。
原文摘要 · Abstract (English)
Recent advancements in Large Language Models (LLMs) have achieved robust performance across diverse tasks, but fine-tuning these models for specific domains remains resource-intensive. Parameter-Efficient Fine-Tuning (PEFT) methods like Low-Rank Adaptation (LoRA) address this challenge by fine-tuning a small subset of parameters. However, existing methods for fusing multiple LoRAs lack dynamic fusion based on contextual inputs and often increase inference time due to token-level operations. We propose DLP-LoRA, a Dynamic Lightweight Plugin that employs a mini-MLP module with only 5M parameters to dynamically fuse multiple LoRAs at the sentence level using top-p sampling strategies. This approach reduces inference time to less than twice that of single LoRA inference by leveraging parallel computation. Evaluations across 26 tasks-including multiple-choice questions and question answering-demonstrate that DLP-LoRA achieves an average accuracy of 92.34% on multiple-choice datasets and significant improvements in BLEU and ROUGE scores on QA datasets, outperforming different LLMs backbones under composite task settings. DLP-LoRA effectively balances performance and efficiency, making it a practical solution for dynamic multi-task adaptation in LLMs. Our code is available at https://github.com/MeCuping/DLP-LoRA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。