用梯度信息初始化向量适配，让小模型也能高效训练
GiVA: Gradient-Informed Bases for Vector-Based Adaptation
- 基于梯度信息设计向量初始化策略，提升训练效率
- 在多个任务上性能优于或媲美LoRA，rank需求降低8倍
- 适合资源受限场景下快速微调大模型
随着模型规模不断增长，参数高效微调已成为全量微调的有力替代方案。尽管LoRA广泛应用，但近期研究探索了参数极省的向量式适配方法。然而，这些方法通常需要远高于LoRA的秩才能达到其性能，导致训练成本上升。本文提出GiVA，一种面向向量式适配的梯度引导初始化策略。该方法实现与LoRA相当的训练时间，同时保持向量式适配的极端参数效率。我们在自然语言理解、生成及图像分类等多样化基准上评估GiVA，实验表明，本方法在各项任务中均显著优于或可媲美现有向量式适配方法及LoRA，且秩需求降低八倍（8×）。
原文摘要 · Abstract (English)
As model sizes continue to grow, parameter-efficient fine-tuning has emerged as a powerful alternative to full fine-tuning. While LoRA is widely adopted among these methods, recent research has explored vector-based adaptation methods due to their extreme parameter efficiency. However, these methods typically require substantially higher ranks than LoRA to match its performance, leading to increased training costs. This work introduces GiVA, a gradient-based initialization strategy for vector-based adaptation. It achieves training times comparable to LoRA and maintains the extreme parameter efficiency of vector-based adaptation. We evaluate GiVA across diverse benchmarks, including natural language understanding, natural language generation, and image classification. Experiments show that our approach consistently outperforms or achieves performance competitive with existing vector-based adaptation methods and LoRA while reducing rank requirements by a factor of eight ($8\times$).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。