用轻量投影网络打通不同文本模态的语义鸿沟，提升跨模态检索效率
Mind the Gap: A Generalized Approach for Cross-Modal Embedding Alignment
- 设计轻量投影网络，统一编码异构文本模态的嵌入表示
- 在跨语言与编程代码等任务上超越BM25和DPR，接近Sentence Transformers精度
- 低资源需求，适合实时、受限环境下的部署应用
检索增强生成（RAG）系统通过引入外部知识提升文本生成质量，但在跨文本模态检索时常因语义差异而失效。本文提出一种基于投影的通用方法，受迁移学习中适配器模块启发，高效弥合编程代码与伪代码、英语与法语句子等异构文本间的语义鸿沟。该方法强调速度、准确率与数据效率，训练与推理仅需极少资源。通过轻量级投影网络将不同文本模态的嵌入对齐至统一空间，模型显著优于传统检索方法如Okapi BM25及Dense Passage Retrieval（DPR），并逼近Sentence Transformers的精度。大量实验验证了方法在多种任务中的有效性与泛化能力，凸显其在实时、资源受限场景下的应用潜力。
原文摘要 · Abstract (English)
Retrieval-Augmented Generation (RAG) systems enhance text generation by incorporating external knowledge but often struggle when retrieving context across different text modalities due to semantic gaps. We introduce a generalized projection-based method, inspired by adapter modules in transfer learning, that efficiently bridges these gaps between various text types, such as programming code and pseudocode, or English and French sentences. Our approach emphasizes speed, accuracy, and data efficiency, requiring minimal resources for training and inference. By aligning embeddings from heterogeneous text modalities into a unified space through a lightweight projection network, our model significantly outperforms traditional retrieval methods like the Okapi BM25 algorithm and models like Dense Passage Retrieval (DPR), while approaching the accuracy of Sentence Transformers. Extensive evaluations demonstrate the effectiveness and generalizability of our method across different tasks, highlighting its potential for real-time, resource-constrained applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。