MiLoRA提升大模型微调效率，降低多租户延迟
MiLoRA: Efficient Mixture of Low-Rank Adaptation for Large Language Models Fine-tuning
- 将LoRA模块视为专家，用提示感知路由机制优化计算
- 首令牌生成后复用路由结果，显著减少推理延迟
- 在多个基准上超越现有方法，适合高并发场景
低秩适配（LoRA）及其混合专家（MOE）变体是高效的参数高效微调（PEFT）方法。然而，在多租户场景下，由于Transformer层中多个线性模块添加了LoRA模块和MOE路由机制，导致显著延迟。为此，我们提出了一种新型高效的LoRA变体——混合低秩适配（MiLoRA）。MiLoRA将每个LoRA模块视为一个专家，并采用提示感知路由机制：该机制在生成首个新标记前计算一次专家路由结果，并在后续标记中复用，从而降低延迟。在常识推理、数学推理任务及广泛使用的LLM评估基准上的大量实验与分析表明，MiLoRA在可比可调参数预算下始终优于强基线。此外，相较于以往基于LoRA的方法，MiLoRA在多租户设置中显著降低了延迟。
原文摘要 · Abstract (English)
Low-rank adaptation (LoRA) and its mixture-of-experts (MOE) variants are highly effective parameter-efficient fine-tuning (PEFT) methods. However, they introduce significant latency in multi-tenant settings due to the LoRA modules and MOE routers added to multiple linear modules in the Transformer layer. To address this issue, we propose Mixture of Low-Rank Adaptation (MiLoRA), a novel and efficient LoRA variant. MiLoRA differs from previous MOE-style LoRA methods by considering each LoRA module as an expert and employing a prompt-aware routing mechanism. This mechanism calculates expert routing results once before generating the first new token and reuses these results for subsequent tokens, reducing latency. Extensive experiments and analysis on commonsense reasoning tasks, math reasoning tasks, and widely used LLM evaluation benchmarks demonstrate that MiLoRA consistently outperforms strong PEFT baselines with comparable tunable parameter budgets. Additionally, MiLoRA significantly reduces latency in multi-tenant settings compared to previous LoRA-based methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。