让多模态嵌入模型智能判断何时增补查询,提升效率与效果。
Let Multimodal Embedders Learn When to Augment Query via Adaptive Query Augmentation
- 基于数据集划分,区分需增补与无需增补的查询
- 仅对必要查询进行增补,嵌入延迟降低显著
- 适用于多模态场景,兼顾速度与检索性能
查询增补通过在查询中附加额外信息来提升其语义,从而更有效地找到相关文档。现有研究采用大语言模型(LLM)构建的嵌入器,以多任务方式联合学习表示与生成能力,实现查询增补与嵌入。推理时,这些联合训练的嵌入器先执行增补再进行嵌入,表现良好。但对所有查询强制增补会带来显著的嵌入延迟,且对部分查询反而损害性能。此外,此前方法未在多模态环境下探索。为此,我们提出 M-Solomon,一种可自适应判断是否增补查询的通用多模态嵌入器。首先,在数据集层面将查询分为两类:需要增补与不需要增补的查询。随后,引入合成过程,利用强大的多模态大模型(MLLM)为需增补的查询生成合适增补内容。最后,提出自适应查询增补机制:M-Solomon 学习为需增补的查询生成前缀 /augment,而对其他查询生成 /embed。实验表明,M-Solomon 不仅大幅超越无增补基线,也优于始终增补的基线,同时嵌入延迟显著降低。
原文摘要 · Abstract (English)
Query augmentation makes queries more meaningful by appending further information to the queries to find relevant documents. Current studies have proposed Large Language Model (LLM)-based embedders, which learn representation for embedding and generation for query augmentation in a multi-task manner by leveraging the generative capabilities of LLM. During inference, these jointly trained embedders have conducted query augmentation followed by embedding, showing effective results. However, augmenting every query leads to substantial embedding latency and query augmentation can be detrimental to performance for some queries. Also, previous methods have not been explored in multimodal environments. To tackle these problems, we propose M-Solomon, a universal multimodal embedder that can adaptively determine when to augment queries. Our approach first divides the queries of the training datasets into two groups at the dataset level. One includes queries that require augmentation and the other includes queries that do not. Then, we introduces a synthesis process that generates appropriate augmentations for queries that require them by leveraging a powerful Multimodal LLM (MLLM). Next, we present adaptive query augmentation. Through this step, M-Solomon can conduct query augmentation only when necessary by learning to generate synthetic augmentations with the prefix /augment for queries that demand them and to generate the simple string /embed for others. Experimental results showed that M-Solomon not only surpassed the baseline without augmentation by a large margin but also outperformed the baseline that always used augmentation, providing much faster embedding latency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。