通过设计嵌入先验，提升提示调优的可解释性与控制力。
Exploring Embedding Priors in Prompt-Tuning for Improved Interpretability and Control
- 引入嵌入先验，改变提示调优中嵌入位置分布。
- 不同任务激活区域分散，但同领域任务聚集于同一空间簇。
- 为思维链蒸馏等任务提供可控的初始化起点。
提示调优是一种高效适应预训练语言模型的方法，通过调整提示嵌入实现最小计算开销。本文研究提示调优中常见的嵌入坍缩现象对模型性能的影响。我们设计了嵌入先验，并与软提示和深度提示调优收敛后的后验进行对比。结果表明，先验显著影响调优后嵌入的位置，模型可在激活空间的不同区域（甚至全新区域）有效工作。由于最终提示调优能力受限，我们推测可控的后验可作为链式思维蒸馏等任务的良好初始化点。实验还发现生成轨迹在激活空间中不集中，但远距离任务（如NLP与算术）形成明显聚类，而同类任务（如问答与掩码语言建模）则位于同一聚类内。这些观察引发对大模型泛化能力是否依赖单一激活簇的思考。
原文摘要 · Abstract (English)
Prompt-Tuning is an efficient method for adapting pre-trained language models to new tasks with minimal computational overhead by modifying prompt embeddings. In this work, we investigate how crucial the phenomenon of embedding collapse, frequently observed in Prompt-Tuning, is for the final performance of the model. To address this question, we designed embedding priors and compared them with posteriors of the converged Soft and Deep Prompt-Tuning methods. Our findings suggest that priors strongly affect the position of the tuned embeddings, and models can effectively work with embeddings from different parts of activation spaces, including completely new regions. As the final Prompt-Tuning capabilities are limited, we hypothesize that controllable Prompt-Tuning posteriors may serve as a good starting point for tasks such as chain-of-thought (COT) distillation. Our experiments also show that generated trajectories are not localized in the activation space of the models. However, there are distinct clusters of activations for distant tasks (e.g., NLP and arithmetic), while activations between NLP tasks (e.g., Question-Answering and MLM) lie in the same cluster. These observations raise questions about the importance of a single activation cluster for the generalization abilities of large language models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。