提升大模型低秩适配训练效率，支持多适配器并发训练
PLoRA: Efficient Concurrent LoRA Training for Large Language Models
- 自动调度多任务并行训练，优化硬件资源利用
- 训练吞吐最高提升12.8倍，整体耗时减少7.52倍
- 适合需要高效微调多个适配器的研究与工业场景
低秩适配（LoRA）因其低资源消耗和良好性能，成为大语言模型微调的热门方法。然而现有研究多聚焦于并发服务多个LoRA适配器的效率，而忽视了训练过程本身资源利用率低、开销高的问题。本文通过大量实证研究发现，当前LoRA训练范式未能有效利用硬件资源，获取高性能适配器代价高昂。为此，我们提出PLoRA，可在给定硬件与模型约束下自动协调多个LoRA训练任务，并开发高效内核提升训练效率。在多种大语言模型与LoRA配置下，PLoRA相较现有方法训练吞吐最高提升12.8倍，整体微调完成时间缩短7.52倍。
原文摘要 · Abstract (English)
Low-Rank Adaptation (LoRA) has gained popularity as a fine-tuning approach for Large Language Models (LLMs) due to its low resource requirements and good performance. While numerous studies have investigated ways to improve LoRA serving efficiency by serving multiple LoRAs concurrently, existing methods assume that a wide range of LoRA adapters are available for serving. In our work, we conduct extensive empirical studies to show that current LoRA training paradigms do not efficiently utilize hardware resources and incur high overhead to obtain a performant LoRA adapter. Leveraging these insights, we propose PLoRA, which automatically orchestrates concurrent LoRA fine-tuning jobs under given hardware and model constraints and develops performant kernels to improve training efficiency. Across a range of LLMs and LoRA configurations, PLoRA improves training throughput by up to 12.8x and reduces the overall fine-tuning makespan by up to 7.52x compared to existing approaches.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。