LoRA改进版通过增强任务相关特征，参数减少33.71%仍更优
Low-Rank Adaptation with Task-Relevant Feature Enhancement for Fine-tuning Language Models
- 引入任务感知滤波器，从隐藏层中提取关键特征
- 在多个任务上参数减少33.71%且性能超越现有低秩方法
- 适合追求高效微调的NLP研究者与工程师
参数高效微调预训练大语言模型受到广泛关注。LoRA 是其中最常用的方法，其假设优化过程本质上是低维的。尽管 LoRA 表现良好，但在学习新任务时仍存在显著性能差距。本文提出低秩适配与任务相关特征增强（LoRATRF），从神经网络表示编辑角度增强任务相关特征。为优先处理任务相关特征，设计了任务感知滤波器，可选择性地从隐藏表示中提取目标任务的有价值知识。在包括自然语言理解、常识推理和数学推理在内的多种数据集上的实验表明，该方法在参数量减少33.71%的情况下，优于当前最先进的低秩微调方法。
原文摘要 · Abstract (English)
Fine-tuning pre-trained large language models in a parameter-efficient manner is widely studied for its effectiveness and efficiency. LoRA is one of the most widely used methods, which assumes that the optimization process is essentially low dimensional. Although LoRA has demonstrated commendable performance, there remains a significant performance gap between LoRA and full fine-tuning when learning new tasks. In this work, we propose Low-Rank Adaptation with Task-Relevant Feature Enhancement(LoRATRF) for enhancing task-relevant features from the perspective of editing neural network representations. To prioritize task-relevant features, a task-aware filter that selectively extracts valuable knowledge from hidden representations for the target or current task is designed. As the experiments on a vareity of datasets including NLU, commonsense reasoning and mathematical reasoning tasks demonstrates, our method reduces 33.71% parameters and achieves better performance on a variety of datasets in comparison with SOTA low-rank methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。