让注意力头共享低秩参数，提升小样本下的模型适配效率。
Hypernetwork-Driven Low-Rank Adaptation Across Attention Heads
- 用超网络统一生成多头注意力的低秩矩阵，实现跨头信息共享。
- 在低数据场景下性能显著优于传统LoRA，样本效率更高。
- 适合资源有限或数据稀缺的下游任务微调，尤其适用于大模型。
参数高效微调（PEFT）已成为在下游任务中以极少额外参数适配大规模预训练模型的有效范式。其中，低秩适配（LoRA）通过插入可训练的低秩矩阵到权重更新中，展现出良好效果。然而，现有基于LoRA的方法在多头自注意力结构中通常独立微调每个注意力头，忽略了头之间的潜在交互与共享结构。为此，本文提出超网络驱动的低秩适配（HyRA），利用超网络为层内所有注意力头生成联合低秩矩阵。共享生成器促进跨头信息共享，帮助低秩模块避免传统LoRA中的冗余特征学习。理论上，该方法相比标准LoRA具有显著更优的样本效率。实验上，在涵盖语言与视觉的广泛基准上评估，HyRA始终优于现有参数高效微调基线。尤其在低数据环境下，其性能显著超越LoRA，凸显其在数据稀缺场景下的实际样本效率与有效性。
原文摘要 · Abstract (English)
Parameter-efficient fine-tuning (PEFT) has emerged as a powerful paradigm for adapting large-scale pre-trained models to downstream tasks with minimal additional parameters. Among PEFT methods, Low-Rank Adaptation (LoRA) stands out for its effectiveness by inserting trainable low-rank matrices into weight updates to enable efficient adaptation. However, when applied to multi-head self-attention, existing LoRA-based methods typically fine-tune each attention head independently, overlooking potential interactions and shared structure among heads. To address this limitation, we propose Hypernetwork-Driven Low-rank Adaptation (HyRA) that employs a hypernetwork to generate joint low-rank matrices for all attention heads within a layer. The shared generator promotes cross-head information sharing, helping low-rank modules avoid the redundant feature learning seen in traditional LoRA methods. Theoretically, our method achieves significantly better sample efficiency compared to standard LoRA. Empirically, we evaluate HyRA on a comprehensive suite of language and vision benchmarks. Our approach consistently outperforms existing parameter-efficient fine-tuning (PEFT) baselines across a wide range of tasks. Notably, in low-data regimes, HyRA achieves substantial improvements over LoRA, underscoring its practical sample efficiency and effectiveness in data-scarce scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。