提出低维可解释文本嵌入LDIR，兼顾语义性能与维度可追踪性。
LDIR: Low-Dimensional Dense and Interpretable Text Embeddings with Relative Representations
- 基于远点采样构建相对锚点文本的语义关联维度
- 500维以下嵌入在相似度/检索任务中接近黑盒模型表现
- 适合需要可解释性的实际部署场景
语义文本表示是自然语言处理的基础任务。现有文本嵌入（如SimCSE和LLM2Vec）性能优异，但各维度数值难以追溯与解释。传统词袋模型虽可解释，但性能较差。近期Benara等人（2024）利用大语言模型生成基于问答的“0/1”可解释嵌入，但通常维度超过10,000。本文提出低维（低于500）、稠密且可解释的相对表示嵌入方法LDIR。其维度数值通过远点采样体现与不同锚定文本的语义相关性，兼具语义表征能力与一定程度的可追踪性与可解释性。我们在多个语义文本相似度、检索与聚类任务上验证LDIR。大量实验结果表明，LDIR性能接近黑盒基线模型，且显著优于其他可解释嵌入基线，同时维度大幅降低。代码已开源：https://github.com/szu-tera/LDIR。
原文摘要 · Abstract (English)
Semantic text representation is a fundamental task in the field of natural language processing. Existing text embedding (e.g., SimCSE and LLM2Vec) have demonstrated excellent performance, but the values of each dimension are difficult to trace and interpret. Bag-of-words, as classic sparse interpretable embeddings, suffers from poor performance. Recently, Benara et al. (2024) propose interpretable text embeddings using large language models, which forms "0/1" embeddings based on responses to a series of questions. These interpretable text embeddings are typically high-dimensional (larger than 10,000). In this work, we propose Low-dimensional (lower than 500) Dense and Interpretable text embeddings with Relative representations (LDIR). The numerical values of its dimensions indicate semantic relatedness to different anchor texts through farthest point sampling, offering both semantic representation as well as a certain level of traceability and interpretability. We validate LDIR on multiple semantic textual similarity, retrieval, and clustering tasks. Extensive experimental results show that LDIR performs close to the black-box baseline models and outperforms the interpretable embeddings baselines with much fewer dimensions. Code is available at https://github.com/szu-tera/LDIR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。