用自注意力动态生成软提示，让大模型更高效适应特定领域任务。
Leveraging Self-Attention for Input-Dependent Soft Prompting in LLMs
- 基于输入内容动态生成软提示，通过自注意力分配不同词元重要性
- 仅需少量可训练参数，实现比现有方法更好的零样本领域迁移能力
- 适合资源有限时提升大模型在垂直领域表现的场景
大语言模型在特定领域任务中表现依赖微调，但微调成本高且技术复杂。本文提出一种新型输入依赖型软提示方法（ID-SPAM），利用自注意力机制根据输入词元生成软提示，并为不同词元分配不同重要性。该方法简单高效，保持可训练参数极少。实验表明，相比当前最优方法，该方法在多个任务上表现更优，显著提升了零样本领域迁移能力。
原文摘要 · Abstract (English)
The performance of large language models in domain-specific tasks necessitates fine-tuning, which is computationally expensive and technically challenging. This paper focuses on parameter-efficient fine-tuning using soft prompting, a promising approach that adapts pre-trained models to downstream tasks by learning a small set of parameters. We propose a novel Input Dependent Soft Prompting technique with a self-Attention Mechanism (ID-SPAM) that generates soft prompts based on the input tokens and attends different tokens with varying importance. Our method is simple and efficient, keeping the number of trainable parameters small. We show the merits of the proposed approach compared to state-of-the-art techniques on various tasks and show the improved zero shot domain transfer capability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。