通过子空间反卷积实现高效微调，参数量减少8倍且性能不降。
Parameter-Efficient Fine-Tuning of Large Language Models via Deconvolution in Subspace
- 用反卷积增强子空间增量矩阵的细节与知识
- 相比LoRA参数减少8倍，仍保持优秀性能
- 适合资源受限场景下的大模型微调
大型语言模型（LLM）被视为迈向通用人工智能（AGI）的重要里程碑。为适配多种下游任务，对LLM进行微调已成为新范式。低秩适应（LoRA）因其参数效率而广受关注，可将微调所需参数量减少数个数量级。然而，基于LoRA的方法受限于秩一分解的瓶颈。随着模型参数规模增大，即使秩一分解也可能超出实际任务所需参数量。本文提出一种基于子空间反卷积的参数高效微调方法（DCFT），创新性地利用反卷积恢复子空间增量矩阵中的细节并增强知识表达，通过动态调整核大小控制参数，摆脱了秩一分解限制。大量实验验证了其有效性：相比LoRA，DCFT在参数量上减少8倍，同时保持优异性能。代码已开源：https://github.com/Godz-z/DCFT。
原文摘要 · Abstract (English)
Large language model (LLM) is considered a milestone towards achieving Artificial General Intelligence (AGI). With its advanced emergent capabilities, it adapt to a wide range of specific applications. Fine-tuning LLMs for various downstream tasks has become a new paradigm. Low-Rank Adaptation (LoRA) is well-known for its parameter efficiency. It can reduce the number of parameters needed to fine-tune LLMs by several orders of magnitude. However, LoRA-based approaches encounter a significant limitation due to the bottleneck imposed by rank one decomposition. As the parameters count in LLMs increase, even rank one decomposition might surpass the number of parameters truly necessary for handling more downstream tasks. In this paper, we propose a new method for Parameter-Efficient Fine-Tuning (PEFT) via deconvolution in subspace, dubbed as DCFT. We innovatively use deconvolution to complete details and enhance knowledge in subspace incremental matrices, and dynamically control parameters by adjusting the kernel size, unconstrained by rank-one decomposition. Extensive experiments are conducted to validate the effectiveness of DCFT. Results show that compared to LoRA, DCFT achieve an 8$\times$ reduction in parameters, and still achieves highly impressive performance. Our code is available here: https://github.com/Godz-z/DCFT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。