先学知识再表征，让大模型在专业领域更懂行
Learn Before Represent: Bridging Generative and Contrastive Learning for Domain-Specific LLM Embeddings
- 先用生成式学习注入领域知识，再做对比学习对齐
- 在医学、化学等任务上显著超越现有方法
- 适合需要精准专业理解的垂直领域应用
通过对比学习微调的大语言模型在通用表征上表现优异，但在化学、法律等垂直领域效果不佳，主要因缺乏领域知识。本文识别出核心瓶颈：当前‘LLM+CL’范式侧重语义对齐，无法实现知识获取，导致难以处理专业术语。为此，提出新型两阶段框架LBR（Learn Before Represent）。第一阶段通过信息瓶颈约束的生成式学习注入领域知识，保留大模型因果注意力以最大化知识获取，同时压缩语义；第二阶段在压缩表示上进行生成式优化的对比学习，实现对齐。该方法保持架构一致性，化解生成与对比学习的目标冲突。在医疗、化学和代码检索任务上的大量实验表明，LBR显著优于强基线模型。本工作建立了垂直领域构建准确鲁棒表征的新范式。
原文摘要 · Abstract (English)
Large Language Models (LLMs) adapted via contrastive learning excel in general representation learning but struggle in vertical domains like chemistry and law, primarily due to a lack of domain-specific knowledge. This work identifies a core bottleneck: the prevailing ``LLM+CL'' paradigm focuses on semantic alignment but cannot perform knowledge acquisition, leading to failures on specialized terminology. To bridge this gap, we propose Learn Before Represent (LBR), a novel two-stage framework. LBR first injects domain knowledge via an Information Bottleneck-Constrained Generative Learning stage, preserving the LLM's causal attention to maximize knowledge acquisition while compressing semantics. It then performs Generative-Refined Contrastive Learning on the compressed representations for alignment. This approach maintains architectural consistency and resolves the objective conflict between generative and contrastive learning. Extensive experiments on medical, chemistry, and code retrieval tasks show that LBR significantly outperforms strong baselines. Our work establishes a new paradigm for building accurate and robust representations in vertical domains.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。