用小波变换压缩词句嵌入，降维93%仍保性能
Semantic Compression for Word and Sentence Embeddings using Discrete Wavelet Transform
- 用离散小波变换分析嵌入向量多尺度特征
- 降维50%-93%后语义相似度几乎不变
- 适配大模型，提升下游任务准确率
小波变换是一种强大的数学工具，广泛应用于信号与图像处理，可揭示复杂模式、增强数据表示并提取有意义特征。实证表明，小波变换可用于自然语言处理，捕捉多种语言与语义特性。本文首次将离散小波变换（DWT）应用于词和句子嵌入，旨在展示其在多分辨率下分析嵌入表示及压缩的能力，同时保持整体质量。我们在语义相似性任务上评估DWT嵌入的有效性，证明其能有效浓缩嵌入向量中的关键语义信息。实验采用多种嵌入模型，包括大语言模型，在下游任务中验证有效性。结果表明，DWT可实现50%-93%的维度压缩，语义相似性任务性能几乎无损；多数下游任务准确率更优。研究为小波变换在自然语言处理中的应用开辟新路径。
原文摘要 · Abstract (English)
Wavelet transforms, a powerful mathematical tool, have been widely used in different domains, including Signal and Image processing, to unravel intricate patterns, enhance data representation, and extract meaningful features from data. Tangible results from their application suggest that Wavelet transforms can be applied to NLP capturing a variety of linguistic and semantic properties. In this paper, we empirically leverage the application of Discrete Wavelet Transforms (DWT) to word and sentence embeddings. We aim to showcase the capabilities of DWT in analyzing embedding representations at different levels of resolution and compressing them while maintaining their overall quality. We assess the effectiveness of DWT embeddings on semantic similarity tasks to show how DWT can be used to consolidate important semantic information in an embedding vector. We show the efficacy of the proposed paradigm using different embedding models, including large language models, on downstream tasks. Our results show that DWT can reduce the dimensionality of embeddings by 50-93% with almost no change in performance for semantic similarity tasks, while achieving superior accuracy in most downstream tasks. Our findings pave the way for applying DWT to improve NLP applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。