随机删掉一半文本嵌入维度，对检索分类影响极小。
Randomly Removing 50% of Dimensions in Text Embeddings has Minimal Impact on Retrieval and Classification Tasks
- 随机移除50%嵌入维度，性能下降不足10%。
- 删掉大量均匀分布的维度反而提升性能。
- 适用于优化模型效率或理解编码机制的研究者。
本文研究了截断文本嵌入对下游任务性能的意外影响。在6个先进文本编码器和26个下游任务中，我们一致发现：随机移除高达50%的嵌入维度，仅导致检索与分类任务性能下降不足10%。考虑到小尺寸嵌入的优势及对文本编码的潜在启示，我们进一步分析该现象。结果表明，这并非因表征空间利用不当所致，反而发现大量均匀分布的维度在被移除后会带来性能提升。这解释了为何平均而言，删除大量维度仍仅造成轻微性能损失。该现象在大语言模型生成任务中用于预测下一个词时也观察到，说明其不局限于分类或检索任务。
原文摘要 · Abstract (English)
In this paper, we study the surprising impact that truncating text embeddings has on downstream performance. We consistently observe across 6 state-of-the-art text encoders and 26 downstream tasks, that randomly removing up to 50% of embedding dimensions results in only a minor drop in performance, less than 10%, in retrieval and classification tasks. Given the benefits of using smaller-sized embeddings, as well as the potential insights about text encoding, we study this phenomenon and find that, contrary to what is suggested in prior work, this is not the result of an ineffective use of representation space. Instead, we find that a large number of uniformly distributed dimensions actually cause an increase in performance when removed. This would explain why, on average, removing a large number of embedding dimensions results in a marginal drop in performance. We make similar observations when truncating the embeddings used by large language models to make next-token predictions on generative tasks, suggesting that this phenomenon is not isolated to classification or retrieval tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。