通过重加权注意力头，用极少量参数实现大模型高效微调。
Data-Efficient Adaptation of LLMs via Attention Head Reweighting
- 仅为每个注意力头学习一个标量，大幅减少可训练参数。
- 在少样本场景下性能超越LoRA，参数量仅为其1/200至1/1000。
- 权重可解释，有助于理解模型的上下文学习机制。
在安全等标签数据稀缺的领域，从有限数据中有效学习至关重要。尽管大语言模型（LLMs）已展现出一定的数据高效学习能力，尤其通过参数高效适配方法，但在处理困难任务的少量样本时仍表现不佳。为此，我们提出注意力头重加权（AHR），一种通过为每个注意力头学习单一标量来适应新文本分类任务的数据高效方法。该方法充分利用了各注意力头的功能专属性，显著减少需学习的参数量。在多个开源文本分类数据集上的实验表明，尽管仅有约0.0001%的模型参数可训练，AHR在少样本条件下仍优于标准基线如LoRA，参数量减少200-1000倍。此外，学习到的权重易于解释，可用于分析模型上下文学习能力背后的注意力头机制。
原文摘要 · Abstract (English)
Learning effectively from limited data is critical in domains like security where labeled examples are scarce. Large language models (LLMs) have demonstrated some capabilities for data-efficient learning, especially through parameter-efficient adaptation methods, but continue to struggle when faced with few samples for difficult tasks. To meet this challenge, we propose Attention Head Reweighting (AHR), a data-efficient method that adapts LLMs to new text-classification tasks by learning only a single scalar per attention head. This drastically reduces the number of parameters that need to be learned by making use of the functional specialization of individual attention heads. Experiments on diverse open-source text classification datasets show that AHR can outperform standard baselines like LoRA when learning from limited samples, despite having 200-1000x fewer trainable parameters, as our AHR only modifies ~0.0001% of the model's parameters. In addition, our learned weights are easy to interpret and can be analyzed to better understand the mechanisms and attention heads responsible for in-context learning abilities in LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。