用生成模型在推理时增强句子嵌入，无需训练即可提升效果。
GASE: Generatively Augmented Sentence Encoding
- 推理时用生成模型改写输入文本，融合原始与生成嵌入。
- 在多个嵌入模型上实现性能提升，低性能模型受益更明显。
- 无需微调，适合资源受限场景下的嵌入优化。
我们提出一种无需训练的方法，通过在推理时利用生成式文本模型进行数据增强，以提升句子嵌入质量。与传统合成训练数据的增强方法不同，该方法不依赖模型参数或微调计算资源。生成增强通过重写、摘要或关键词提取等方式变异输入文本，随后对原始与合成嵌入进行聚合。在大规模语义文本相似度基准（Massive Text Embedding Benchmark for STS）上的实验表明，使用不同生成模型进行增强后，多种嵌入模型均获得性能提升。研究发现，基础性能较低的嵌入模型在生成增强下获益更大。结果表明，推理时引入生成增强可增加语义多样性，提升嵌入的鲁棒性与泛化能力。性能增益取决于嵌入模型和数据集。
原文摘要 · Abstract (English)
We propose a training-free approach to improve sentence embeddings leveraging test-time compute by applying generative text models for data augmentation at inference time. Unlike conventional data augmentation that utilises synthetic training data, our approach does not require access to model parameters or the computational resources typically required for fine-tuning state-of-the-art models. Generatively Augmented Sentence Encoding variates the input text by paraphrasing, summarising, or extracting keywords, followed by pooling the original and synthetic embeddings. Experimental results on the Massive Text Embedding Benchmark for Semantic Textual Similarity (STS) demonstrate performance improvements across a range of embedding models using different generative models for augmentation. We find that generative augmentation leads to larger performance improvements for embedding models with lower baseline performance. These findings suggest that integrating generative augmentation at inference time adds semantic diversity and can enhance the robustness and generalisability of sentence embeddings for embedding models. Our results show that performance gains depend on the embedding model and the dataset.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。