通过分解与压缩外积,让提示调优更高效且表现更强。
Efficient and Effective Prompt Tuning via Prompt Decomposition and Compressed Outer Product
- 用截断SVD降低提示参数维度,减少训练开销。
- 通过压缩外积增强提示词间语义交互,提升模型理解力。
- 平均池化压缩内存占用,适合资源受限场景使用。
提示调优(PT)为大规模预训练语言模型提供了一种低成本替代微调的方法,仅需在输入前添加少量软提示参数。然而现有方法存在两大问题:(i) 忽视软提示词间的内在语义关联,导致提示离散性高、交互有限,影响复杂任务表现;(ii) 为应对下游任务复杂性,需使用长提示以提升性能,但提示长度与内存和计算成本正相关。为此,我们提出低参数提示调优(LAMP),结合提示分解与压缩外积机制。提示分解模块采用截断SVD降低软提示参数空间维度;压缩外积模块促进提示词间多层级交互,挖掘其内在关联以增强知识表示;最后通过平均池化减少内存占用与训练/推理时间。在六种架构、八个数据集上的实验表明,LAMP在性能与效率上均优于当前最先进的基于PT和LoRA的方法。
原文摘要 · Abstract (English)
Prompt tuning (PT) offers a cost-effective alternative to fine-tuning large-scale pre-trained language models (PLMs), requiring only a few parameters in soft prompt tokens added before the input text. However, existing PT approaches face two significant issues: (i) They overlook intrinsic semantic associations between soft prompt tokens, leading to high discreteness and limited interactions, thus reducing the model's comprehension and effectiveness in complex tasks. (ii) Due to the complexity of downstream tasks, long soft prompt is necessitated to improve performance, but prompt length correlates positively with memory usage and computational costs. Achieving high efficiency and performance remains an ongoing challenge. To address these issues, we propose a novel Low-parameters prompt tuning (LAMP) method, which leverages prompt decomposition and compressed outer product. Specifically, the prompt decomposition module employs Truncated SVD to reduce training parameters and significantly lower the dimensionality of the soft prompt parameter space. It then utilizes a compressed outer product module to facilitate multiple interactions among prompt tokens, exploring their intrinsic associations to enhance knowledge representation. Finally, LAMP uses average pooling to reduce memory usage and training/inference time. Extensive experiments across six architectures and eight datasets demonstrate that LAMP outperforms state-of-the-art PT-based and LoRA-based methods in performance and efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。