用共享潜变量让冻结的多模态模型高效检索，不破坏原有知识结构。
SLQ: Bridging Modalities via Shared Latent Queries for Retrieval with Frozen MLLMs
- 引入共享潜变量融合图文信息，无需修改主干模型参数。
- 在COCO和Flickr30K上超越全微调与LoRA，KARR-Bench提升显著。
- 适合希望保留预训练能力又需适配检索任务的研究者。
多模态大语言模型具备内在推理与世界知识能力，但将其用于密集检索仍具挑战。现有方法依赖侵入式参数更新（如全微调和LoRA），可能破坏预训练语义空间，影响推理所需结构化知识。为此，我们提出SLQ——一种参数高效微调框架，使冻结的MLLM适应检索任务。SLQ引入少量共享潜变量，分别附加于文本与图像标记，利用模型原生因果注意力机制，将多模态上下文聚合至统一嵌入空间。此外，为更全面评估检索能力，我们构建KARR-Bench基准，用于知识感知推理检索。大量实验表明，SLQ在COCO与Flickr30K上优于全微调与LoRA，MMEB表现相当，并在KARR-Bench上取得显著提升，验证了非侵入式适配保留预训练表征的有效性。代码已开源：https://github.com/CnFaker/SLQ。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) possess intrinsic reasoning and world-knowledge capabilities, yet adapting them for dense retrieval remains challenging. Existing approaches rely on invasive parameter updates, such as full fine-tuning and LoRA, which may disrupt the pre-trained semantic space and impair the structured knowledge essential for reasoning. To address this, we propose SLQ, a parameter-efficient tuning framework that adapts MLLMs for retrieval while keeping the backbone entirely frozen. SLQ introduces a small set of Shared Latent Queries that are appended to both text and image tokens, leveraging the model's native causal attention to aggregate multimodal context into a unified embedding space. Furthermore, to better evaluate retrieval beyond superficial pattern matching, we construct KARR-Bench, a benchmark designed for knowledge-aware reasoning retrieval. Extensive experiments show that SLQ outperforms full fine-tuning and LoRA on COCO and Flickr30K, while achieving competitive performance on MMEB and yielding substantial gains on KARR-Bench, validating that preserving the pre-trained representations via non-invasive adaptation is an effective strategy for MLLM-based retrieval. The code is available under: https://github.com/CnFaker/SLQ.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。