用低秩适配器改进注意力机制，提升大模型微调效果。
DiffLoRA: Differential Low-Rank Adapters for Large Language Models
- 在正负注意力项上加入低秩适配器，实现参数高效微调。
- 在HumanEval任务上比LoRA高出11分，其他任务表现接近。
- 适合关注代码生成等特定任务的开发者使用。
近期提出的差分Transformer通过去噪注意力机制提升Transformer模型性能。本文提出DiffLoRA，将差分注意力机制与低秩适配器结合，在正负注意力项上均引入低秩适配器，兼顾效率与性能。我们在涵盖通用基准、少样本上下文学习、检索增强生成和长文本测试的多种NLP任务中评估该方法。结果表明，尽管在多数任务上不及其他参数高效微调方法，但在部分领域表现突出——如HumanEval任务上较LoRA提升11分。我们分析了微调后注意力模式，揭示其表现差异原因。
原文摘要 · Abstract (English)
Differential Transformer has recently been proposed to improve performance in Transformer models by canceling out noise through a denoiser attention mechanism. In this work, we introduce DiffLoRA, a parameter-efficient adaptation of the differential attention mechanism, with low-rank adapters on both positive and negative attention terms. This approach retains the efficiency of LoRA while aiming to benefit from the performance gains of differential attention. We evaluate DiffLoRA across a broad range of NLP tasks, including general benchmarks, many-shot in-context learning, RAG, and long-context tests. We observe that, although DiffLoRA falls short of other parameter-efficient fine-tuning methods in most evaluation tasks, it shows interesting results in certain domains (+11 pts on LoRA for HumanEval). We analyze the attention patterns post-finetuning to identify the reasons for this behavior.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。