UORA通过正交重初始化实现高效微调，参数更少、性能更强。
UORA: Uniform Orthogonal Reinitialization Adaptation in Parameter-Efficient Fine-Tuning of Large Models
- 用插值重参数化选择性重置冻结矩阵的行与列
- 相比LoRA参数减少70%以上，计算存储效率更高
- 适合资源受限场景下的大模型微调
本文提出一种新型参数高效微调方法UORA，用于大语言模型。UORA通过低秩近似大幅减少可训练参数数量，采用基于插值的重参数化机制，依据向量幅度启发式选择性重置冻结投影矩阵中的行和列。相比LoRA，其可训练参数显著减少；在计算与存储效率上优于VeRA。在GLUE和E2E等多类基准测试中表现优异，适用于大模型指令微调及图像分类任务。实验验证了其在保持高性能的同时，带来可忽略的计算开销，为大模型的可扩展、资源高效微调提供了新范式。
原文摘要 · Abstract (English)
This paper introduces Uniform Orthogonal Reinitialization Adaptation (UORA), a novel parameter-efficient fine-tuning (PEFT) approach for Large Language Models (LLMs). UORA achieves state-of-the-art performance and parameter efficiency by leveraging a low-rank approximation method to reduce the number of trainable parameters. Unlike existing methods such as LoRA and VeRA, UORA employs an interpolation-based reparametrization mechanism that selectively reinitializes rows and columns in frozen projection matrices, guided by the vector magnitude heuristic. This results in substantially fewer trainable parameters compared to LoRA and outperforms VeRA in computation and storage efficiency. Comprehensive experiments across various benchmarks demonstrate UORA's superiority in achieving competitive fine-tuning performance with negligible computational overhead. We demonstrate its performance on GLUE and E2E benchmarks and its effectiveness in instruction-tuning large language models and image classification models. Our contributions establish a new paradigm for scalable and resource-efficient fine-tuning of LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。