用任务专家混合提升嵌入模型性能，不改参数量也不变推理速度。
Beyond instruction-conditioning, MoTE: Mixture of Task Experts for Multi-task Embedding Models
- 设计任务专家混合模块，通过任务感知对比学习训练专用参数。
- 在检索数据集上性能提升64%，全数据集提升43%。
- 适合需要高效多任务嵌入的工业级应用，无需额外计算开销。
密集嵌入是现代机器学习系统的核心，支撑检索增强生成、信息检索和表示学习。尽管指令调制已成为嵌入专化的主流方法，但其直接应用于低容量模型会带来根本性的表征限制，制约了专业化带来的性能提升。本文分析了这些局限，并提出任务专家混合（MoTE）Transformer模块，利用任务感知对比学习（TA-CL）训练的任务专用参数，增强模型生成专业化嵌入的能力。实验结果表明，MoTE在检索数据集上性能提升64%（+3.27 → +5.21），所有数据集上提升43%（+1.81 → +2.60）。关键的是，这些增益无需修改指令、训练数据、推理时间或活跃参数数量即可实现。
原文摘要 · Abstract (English)
Dense embeddings are fundamental to modern machine learning systems, powering Retrieval-Augmented Generation (RAG), information retrieval, and representation learning. While instruction-conditioning has become the dominant approach for embedding specialization, its direct application to low-capacity models imposes fundamental representational constraints that limit the performance gains derived from specialization. In this paper, we analyze these limitations and introduce the Mixture of Task Experts (MoTE) transformer block, which leverages task-specialized parameters trained with Task-Aware Contrastive Learning (\tacl) to enhance the model ability to generate specialized embeddings. Empirical results show that MoTE achieves $64\%$ higher performance gains in retrieval datasets ($+3.27 \rightarrow +5.21$) and $43\%$ higher performance gains across all datasets ($+1.81 \rightarrow +2.60$). Critically, these gains are achieved without altering instructions, training data, inference time, or number of active parameters.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。