用双模型框架让文本嵌入更高效且跨架构通用
PromptEmbedder: Efficient and Transferable Text Embedding via Dual-LLM Soft Prompting
- 用两个大模型分工：一个生成提示，一个固定嵌入
- 比传统方法省40%显存，训练快3.7倍，性能相当
- 换新模型只需调小矩阵，适合快速部署和迁移
大型语言模型在文本嵌入方面表现优异，但现有适配方法如LoRA在计算效率和跨架构迁移性上存在瓶颈，每当出现新骨干网络时需从头重新训练。为此，我们提出PromptEmbedder，一种新型双大模型框架，将嵌入知识与特定骨干权重解耦。PromptEmbedder利用提示生成模型通过可微生成过程与连续松弛，为冻结的嵌入模型生成指令感知的软提示，确保对比训练中全程梯度流动。通过将任务相关知识集中于提示生成模型，适应新架构仅需重新训练轻量级线性对齐矩阵。在MTEB基准上的评估显示，PromptEmbedder在性能上与LoRA微调相当，同时显存降低40%，训练速度提升3.7倍。该方法建立了一种可扩展、架构无关的高效大模型表示学习范式。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have demonstrated remarkable efficacy in text embedding, yet current adaptation methods like LoRA face significant bottlenecks in computational efficiency and cross-architecture transferability. Whenever a new backbone emerges, existing approaches require costly retraining from scratch. To address this, we propose PromptEmbedder, a novel dual-LLM framework that decouples embedding knowledge from specific backbone weights. PromptEmbedder utilizes a Prompting LLM to generate instruction-aware soft prompts for a frozen Embedding LLM via a differentiable generation process with continuous relaxation, ensuring full gradient flow during contrastive training. By localizing task-specific knowledge within the Prompting LLM, adapting to new architectures requires only retraining a lightweight linear alignment matrix. Evaluations on the MTEB benchmark show that PromptEmbedder achieves comparable performance with LoRA finetuning while reducing GPU memory by 40% and accelerating training by 3.7x. Our approach establishes a scalable, architecture-agnostic paradigm for efficient LLM-based representation learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。