通过线性分解模型解析多模态多语言句子嵌入，揭示编码器偏见。
FLiP: Towards understanding and interpreting multimodal multilingual sentence embeddings
- 用因子化线性投影从嵌入中还原词汇内容。
- 在多种语言和模型上还原率超75%，优于非因子化方法。
- 无需下游任务即可洞察编码器的模态与语言偏见。
本文提出因子化线性投影（FLiP）模型，用于理解预训练句子嵌入空间。我们在高资源和中资源语言中，对多语言（LaBSE）、多模态（SONAR）及基于API的（Gemini）句子嵌入空间训练了FLiP模型，以恢复其词汇内容。结果表明，FLiP能从嵌入中还原超过75%的词汇信息，显著优于现有非因子化基线方法。以此为诊断工具，我们揭示了所选句子编码器在模态和语言层面的偏差，并为从业者提供了不依赖传统下游评估任务的内在洞察。代码已公开于 https://github.com/BUTSpeechFIT/FLiP。
原文摘要 · Abstract (English)
This paper presents factorized linear projection (FLiP) models for understanding pretrained sentence embedding spaces. We train FLiP models to recover the lexical content from multilingual (LaBSE), multimodal (SONAR) and API-based (Gemini) sentence embedding spaces in several high- and mid-resource languages. We show that FLiP can recall more than 75% of lexical content from the embeddings, significantly outperforming existing non-factorized baselines. Using this as a diagnostic tool, we uncover the modality and language biases across the selected sentence encoders and provide practitioners with intrinsic insights about the encoders without relying on conventional downstream evaluation tasks. Our implementation is public https://github.com/BUTSpeechFIT/FLiP.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。