通过微调与剪枝，让低资源生成任务中的LoRA模块更灵活适配。
Adaptive LoRA Merge with Parameter Pruning for Low-Resource Generation
- 在少量数据下微调并剪枝LoRA参数，实现更精细调整。
- 跨语言多领域摘要任务中，性能显著优于现有方法。
- 适合低资源场景下的大模型轻量适配，如小语种生成。
本研究提出一种简单而有效的LoRA合并方法，用于低资源语言生成任务的大模型适配。现有的LoRA合并技术虽能整合多个任务训练的LoRA模块，被视为高效适配大模型的有效方式，但以往方法因冻结LoRA参数而适应性受限，且未覆盖低资源问题。本文提出的合并方法在极少量目标任务数据下进行微调，并动态剪枝LoRA参数，实现更细粒度的参数调节，提升任务适配能力。实验以摘要任务为基准，在涵盖多种领域和英、日双语的多个数据集上验证，结果表明，该方法在任务适应性上显著且一致地优于已有方法。
原文摘要 · Abstract (English)
This study proposes a simple yet effective LoRA merge method to achieve LLM adaptation for low-resource language generation tasks. The LoRA merge technique, which integrates multiple LoRA modules trained on different tasks, has gained attention as an effective and efficient approach for adapting LLMs to target tasks. However, previous methods are limited in adaptability as they keep the LoRA parameters frozen. Additionally, the low-resource problem has been out of their scope. We propose a LoRA merge method that updates and prunes LoRA parameters through fine-tuning with minimal target task data, which allows finer-grained adjustments of LoRA parameters and enhancement of task adaptability. Extensive experiments have been conducted taking summarization as a benchmark task. Our datasets cover various domains and multiple languages of English and Japanese. The results confirm that the proposed method achieves significant and consistent improvements in task adaptability over the previous methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。