让句子嵌入自动适应上下文，提升语义相似度判断准确率
CASE -- Condition-Aware Sentence Embeddings for Conditional Semantic Textual Similarity Measurement
- 用大模型编码条件，让句子影响上下文注意力分布
- 减去条件嵌入使向量空间更均匀，显著提升任务表现
- 小维度投影也能大幅改进现有模型，适合快速部署
句子的语义常依赖于其上下文。尽管句嵌入方法已有进展，如何根据上下文有效调整嵌入仍不明确。为此，我们提出条件感知句嵌入（CASE），一种高效准确的条件化句嵌入方法。首先，利用大语言模型（LLM）编码器生成条件嵌入，其中句子通过池化过程影响条件中各词元的注意力得分。随后，通过监督学习将基于LLM的文本嵌入对齐到条件语义文本相似度（C-STS）任务。我们发现，减去条件嵌入可稳定提升基于LLM的嵌入在C-STS上的性能，改善嵌入空间的各向同性。此外，我们的监督投影方法在仅需少量嵌入维度的情况下，显著提升现有模型表现。
原文摘要 · Abstract (English)
The meaning conveyed by a sentence often depends on the context in which it appears. Despite the progress of sentence embedding methods, it remains unclear as how to best modify a sentence embedding conditioned on its context. To address this problem, we propose Condition-Aware Sentence Embeddings (CASE), an efficient and accurate method to create an embedding for a sentence under a given condition. First, CASE creates an embedding for the condition using a Large Language Model (LLM) encoder, where the sentence influences the attention scores computed for the tokens in the condition during pooling. Next, a supervised method is learnt to align the LLM-based text embeddings with the Conditional Semantic Textual Similarity (C-STS) task. We find that subtracting the condition embedding consistently improves the C-STS performance of LLM-based text embeddings by improving the isotropy of the embedding space. Moreover, our supervised projection method significantly improves the performance of LLM-based embeddings despite requiring a small number of embedding dimensions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。