通过对比提示提升大模型句子嵌入质量，无需微调即可增强语义表达。
Contrastive Prompting Enhances Sentence Embeddings in LLMs through Inference-Time Steering
- 引入辅助提示进行对比，引导模型聚焦核心语义而非冗余词。
- 在多个任务上显著提升现有提示方法的性能，跨模型通用。
- 即插即用，适用于不同大模型和下游应用，部署简单。
从大语言模型（LLMs）中提取句子嵌入是一种实用方法，无需额外数据或微调。以往研究主要通过提示工程引导模型将句子的核心语义编码到最后一层的嵌入中，但该嵌入仍包含大量非必要信息（如停用词），限制了其编码能力。为此，我们提出一种对比提示（Contrastive Prompting, CP）方法，引入一个额外的辅助提示，通过与之对比，引导现有提示更准确地编码句子核心语义，而非无关信息。CP是一种即插即用的推理时干预方法，可与多种基于提示的方法结合。在语义文本相似性（STS）任务和下游分类任务上的大量实验表明，该方法能有效提升不同大模型上已有提示方法的性能。代码将公开于 https://github.com/zifengcheng/CP。
原文摘要 · Abstract (English)
Extracting sentence embeddings from large language models (LLMs) is a practical direction, as it requires neither additional data nor fine-tuning. Previous studies usually focus on prompt engineering to guide LLMs to encode the core semantic information of the sentence into the embedding of the last token. However, the last token in these methods still encodes an excess of non-essential information, such as stop words, limiting its encoding capacity. To this end, we propose a Contrastive Prompting (CP) method that introduces an extra auxiliary prompt to elicit better sentence embedding. By contrasting with the auxiliary prompt, CP can steer existing prompts to encode the core semantics of the sentence, rather than non-essential information. CP is a plug-and-play inference-time intervention method that can be combined with various prompt-based methods. Extensive experiments on Semantic Textual Similarity (STS) tasks and downstream classification tasks demonstrate that our method can improve the performance of existing prompt-based methods across different LLMs. Our code will be released at https://github.com/zifengcheng/CP.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。