Astra通过激活空间尾部特征向量提升大模型微调效率与性能
Astra: Activation-Space Tail-Eigenvector Low-Rank Adaptation of Large Language Models
- 利用任务校准集估计输出激活的尾部特征向量，构建适配任务的低秩适配器
- 在16个基准上均优于现有高效微调方法，部分场景超越全量微调
- 参数量极低，收敛更快，适合资源受限场景下的模型精调
参数高效微调（PEFT）方法，尤其是LoRA，因计算和存储效率高而被广泛用于将预训练模型适配到下游任务。然而，在LoRA及其变体中，对应于尾部特征向量的激活子空间潜力尚未得到充分利用，可能导致微调性能不理想。本文提出Astra（激活空间尾部特征向量低秩适配），一种新型PEFT方法，通过小规模任务特异性校准集估计模型输出激活的尾部特征向量，构建任务自适应的低秩适配器。通过将更新限制在这些尾部特征向量张成的子空间内，Astra实现了更快的收敛速度和更优的下游性能，且参数预算显著降低。在自然语言理解（NLU）和自然语言生成（NLG）任务上的大量实验表明，Astra在16个基准上均持续优于现有PEFT基线，甚至在某些场景下超越全量微调（FFT）。
原文摘要 · Abstract (English)
Parameter-Efficient Fine-Tuning (PEFT) methods, especially LoRA, are widely used for adapting pre-trained models to downstream tasks due to their computational and storage efficiency. However, in the context of LoRA and its variants, the potential of activation subspaces corresponding to tail eigenvectors remains substantially under-exploited, which may lead to suboptimal fine-tuning performance. In this work, we propose Astra (Activation-Space Tail-Eigenvector Low-Rank Adaptation), a novel PEFT method that leverages the tail eigenvectors of the model output activations-estimated from a small task-specific calibration set-to construct task-adaptive low-rank adapters. By constraining updates to the subspace spanned by these tail eigenvectors, Astra achieves faster convergence and improved downstream performance with a significantly reduced parameter budget. Extensive experiments across natural language understanding (NLU) and natural language generation (NLG) tasks demonstrate that Astra consistently outperforms existing PEFT baselines across 16 benchmarks and even surpasses full fine-tuning (FFT) in certain scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。