MoE模型的路由机制可直接当嵌入模型用，无需微调
Your Mixture-of-Experts LLM Is Secretly an Embedding Model For Free
- 利用MoE模型的专家路由权重生成嵌入向量
- 在20个数据集上比传统隐藏状态嵌入更鲁棒、语义更准确
- 适合想快速获得高质量嵌入但不想微调的开发者
尽管大型语言模型（LLM）在生成任务中表现优异，但其仅解码器架构通常限制了其作为嵌入模型的潜力，除非进行额外表示微调。为回答这一问题，我们深入研究了混合专家（MoE）LLM。研究表明，MoE中的专家路由机制可作为无需微调的即插即用嵌入模型，在多样化嵌入任务中表现良好。我们的分析显示，路由权重（RW）与隐藏状态（HS）互补：相比HS，RW对提示选择更鲁棒，更关注高层语义。受此启发，我们提出MoEE，融合RW与HS，性能优于单独使用任一方式。实验在大规模文本嵌入基准（MTEB）的6个任务、20个数据集上进行，结果表明MoEE显著提升了基于LLM的嵌入效果，且无需进一步微调。
原文摘要 · Abstract (English)
While large language models (LLMs) excel on generation tasks, their decoder-only architecture often limits their potential as embedding models if no further representation finetuning is applied. Does this contradict their claim of generalists? To answer the question, we take a closer look at Mixture-of-Experts (MoE) LLMs. Our study shows that the expert routers in MoE LLMs can serve as an off-the-shelf embedding model with promising performance on a diverse class of embedding-focused tasks, without requiring any finetuning. Moreover, our extensive analysis shows that the MoE routing weights (RW) is complementary to the hidden state (HS) of LLMs, a widely-used embedding. Compared to HS, we find that RW is more robust to the choice of prompts and focuses on high-level semantics. Motivated by the analysis, we propose MoEE combining RW and HS, which achieves better performance than using either separately. Our exploration of their combination and prompting strategy shed several novel insights, e.g., a weighted sum of RW and HS similarities outperforms the similarity on their concatenation. Our experiments are conducted on 6 embedding tasks with 20 datasets from the Massive Text Embedding Benchmark (MTEB). The results demonstrate the significant improvement brought by MoEE to LLM-based embedding without further finetuning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。