系统梳理LoRA变体，统一方法与评估标准。
A Unified Study of LoRA Variants: Taxonomy, Review, Codebase, and Empirical Evaluation
- 按秩、优化动态等四维度分类变体，建立统一理论框架。
- 实验证明合理调参下LoRA性能优于多数变体。
- 开源模块化代码库支持快速实验与分析，适合研究者使用。
低秩适配（LoRA）是一种在大规模神经网络中平衡效率与性能的参数高效微调方法。然而，LoRA变体的大量涌现导致方法、理论、代码和评估的碎片化。为此，本文首次提出对LoRA变体的统一研究，涵盖系统性分类、统一理论综述、结构化代码库及标准化实证评估。首先，我们沿秩、优化动态、初始化及与专家混合（Mixture-of-Experts）集成四个核心维度对变体进行分类；其次，在统一的低秩更新动力学理论框架下回顾其关系与演进；进一步，提出LoRAFactory，一个模块化代码库，通过统一接口实现变体，支持即插即用实验与细粒度分析。最后，基于该代码库，在自然语言生成、自然语言理解与图像分类任务上开展大规模评估，系统探索关键超参数。结果揭示：相比其他超参数，LoRA及其变体对学习率选择极为敏感；且在恰当超参数配置下，LoRA始终达到或超越多数变体性能。
原文摘要 · Abstract (English)
Low-Rank Adaptation (LoRA) is a fundamental parameter-efficient fine-tuning method that balances efficiency and performance in large-scale neural networks. However, the proliferation of LoRA variants has led to fragmentation in methodology, theory, code, and evaluation. To this end, this work presents the first unified study of LoRA variants, offering a systematic taxonomy, unified theoretical review, structured codebase, and standardized empirical assessment. First, we categorize LoRA variants along four principal axes: rank, optimization dynamics, initialization, and integration with Mixture-of-Experts. Then, we review their relationships and evolution within a common theoretical framework focused on low-rank update dynamics. Further, we introduce LoRAFactory, a modular codebase that implements variants through a unified interface, supporting plug-and-play experimentation and fine-grained analysis. Last, using this codebase, we conduct a large-scale evaluation across natural language generation, natural language understanding, and image classification tasks, systematically exploring key hyperparameters. Our results uncover several findings, notably: LoRA and its variants exhibit pronounced sensitivity to the choices of learning rate compared to other hyperparameters; moreover, with proper hyperparameter configurations, LoRA consistently matches or surpasses the performance of most of its variants.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。