复用已有适配器，快速高效实现模型微调。
EigenLoRAx: Recycling Adapters to Find Principal Subspaces for Resource-Efficient Adaptation and Inference
- 复用预训练适配器构建主子空间，降低参数量
- 仅需学习子空间系数，大幅减少训练开销
- 适合边缘设备、个人化部署等低资源场景
大模型的快速发展带来了显著的计算成本，引发环境影响与可及性问题。低秩适配器（LoRA）提供了轻量级微调方案，已积累大量面向不同领域的公开适配器。我们提出EigenLoRAx，通过复用现有适配器构建共享领域知识的主子空间，并在资源受限时添加正交基向量。新任务可通过仅学习子空间主成分的轻量系数快速适应，无需微调完整适配器。该方法显著减少参数与内存占用，提升训练与推理效率。在多样任务与领域上表现优异，适用于边缘应用、个性化及资源受限环境下的大模型公平部署。
原文摘要 · Abstract (English)
The rapid growth of large models has raised concerns about their environmental impact and equity in accessibility due to significant computational costs. Low-Rank Adapters (LoRA) offer a lightweight solution for finetuning large models, resulting in an abundance of publicly available adapters tailored to diverse domains. We ask: Can these pretrained adapters be leveraged to further streamline adaptation to new tasks while addressing these challenges? We introduce EigenLoRAx, a parameter-efficient finetuning method that recycles existing adapters to create a principal subspace aligned with their shared domain knowledge which can be further augmented with orthogonal basis vectors in low-resource scenarios. This enables rapid adaptation to new tasks by learning only lightweight coefficients on the principal components of the subspace-eliminating the need to finetune entire adapters. EigenLoRAx requires significantly fewer parameters and memory, improving efficiency for both training and inference. Our method demonstrates strong performance across diverse domains and tasks, offering a scalable for edge-based applications, personalization, and equitable deployment of large models in resource-constrained environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。