对比专用大模型在密集检索中的表现,发现数学与代码专长影响检索效果。
A Comparative Study of Specialized LLMs as Dense Retrievers
- 用8个不同专长的Qwen2.5 7B模型测试文本、代码、多模态检索能力
- 数学与长推理模型在三类任务中均表现下降,说明推理与语义匹配冲突
- 代码与视觉语言模型零样本表现优异,超越BM25,适合多模态统一检索
尽管大语言模型(LLMs)越来越多地被用作密集检索器,但其领域特化对检索效果的影响仍缺乏深入研究。本研究系统考察了任务特定适配对LLM检索能力的影响,为构建能处理文本、代码、图像及多模态内容的统一检索器奠定基础。我们在零样本设置下,使用8个Qwen2.5 7B LLMs(包括基础模型、指令微调、代码/数学专精、长推理和视觉语言模型),在BEIR(文本检索)和CoIR(代码检索)基准上进行评估;在监督设置下,所有模型均在MS MARCO数据集上微调。结果表明,数学专精与长推理能力在三种设置中均导致性能持续下降,揭示了数学推理与语义匹配之间的冲突。视觉语言模型和代码专精模型在零样本下表现优于其他模型,甚至在代码检索任务中超越BM25,在监督设置中与基础模型性能相当。这些发现为利用跨领域与跨模态融合实现统一检索提供了新方向。
原文摘要 · Abstract (English)
While large language models (LLMs) are increasingly deployed as dense retrievers, the impact of their domain-specific specialization on retrieval effectiveness remains underexplored. This investigation systematically examines how task-specific adaptations in LLMs influence their retrieval capabilities, an essential step toward developing unified retrievers capable of handling text, code, images, and multimodal content. We conduct extensive experiments with eight Qwen2.5 7B LLMs, including base, instruction-tuned, code/math-specialized, long reasoning, and vision-language models across zero-shot retrieval settings and the supervised setting. For the zero-shot retrieval settings, we consider text retrieval from the BEIR benchmark and code retrieval from the CoIR benchmark. Further, to evaluate supervised performance, all LLMs are fine-tuned on the MS MARCO dataset. We find that mathematical specialization and the long reasoning capability cause consistent degradation in three settings, indicating conflicts between mathematical reasoning and semantic matching. The vision-language model and code-specialized LLMs demonstrate superior zero-shot performance compared to other LLMs, even surpassing BM25 on the code retrieval task, and maintain comparable performance to base LLMs in supervised settings. These findings suggest promising directions for the unified retrieval task leveraging cross-domain and cross-modal fusion.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。