用可学习的代码本共享提示向量，仅调0.3%参数就达到顶尖性能。
ACCEPT: Adaptive Codebook for Composite and Efficient Prompt Tuning
- 用产品量化思想设计共享代码本，通过自适应权重区分不同提示。
- 在17个任务上仅更新0.3%参数即达到先进水平，少样本和大模型表现佳。
- 适合资源有限但需高效微调大模型的研究者或部署场景。
Prompt Tuning 是一种参数高效微调方法，因其在大规模预训练语言模型（PLMs）上以极少参数更新即可实现优异性能而受到关注。传统方法将每个提示视为不可分割且独立更新，导致参数随提示长度线性增长。为此，我们提出自适应代码本复合高效提示调优（ACCEPT）。该方法借鉴产品量化（PQ）思想，在每个子空间中让所有软提示共享一组可学习的代码本向量，每个提示通过一组自适应权重进行区分。在17个涵盖自然语言理解（NLU）和问答（QA）的多样化任务上，仅调整PLMs 0.3%的参数，便实现了卓越性能。本方法在少样本和大模型设置下同样表现出色，展现出巨大潜力。
原文摘要 · Abstract (English)
Prompt Tuning has been a popular Parameter-Efficient Fine-Tuning method attributed to its remarkable performance with few updated parameters on various large-scale pretrained Language Models (PLMs). Traditionally, each prompt has been considered indivisible and updated independently, leading the parameters increase proportionally as prompt length grows. To address this issue, we propose Adaptive Codebook for Composite and Efficient Prompt Tuning (ACCEPT). In our method, we refer to the concept of product quantization (PQ), allowing all soft prompts to share a set of learnable codebook vectors in each subspace, with each prompt differentiated by a set of adaptive weights. We achieve the superior performance on 17 diverse natural language tasks including natural language understanding (NLU) and question answering (QA) tasks by tuning only 0.3% of parameters of the PLMs. Our approach also excels in few-shot and large model settings, highlighting its significant potential.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。