动态预测视觉检索的插值权重,提速降耗。
Learning Sample-wise Rank-aware Interpolation Weights for Composed Visual Data Retrieval

- 基于嵌入空间线性插值,动态生成查询专属权重
- 在视频检索上达到最优,图像检索持平顶尖水平
- 通过记忆库合成难负样本,显著降低查询延迟
复合视觉数据检索的核心在于将参考视觉输入与文本修改融合为单一查询。现有最先进方法虽利用多模态大语言模型完成融合,但其复杂性导致查询延迟过高,难以扩展。本文重新审视嵌入空间中简单线性插值的有效性,提出首个动态预测查询特定插值权重的框架SRAIN。关键挑战在于:插值权重质量需由插值后嵌入向量对负样本的区分度及与真实目标的接近程度共同决定,这使得最优权重的收集与预测不可行。为此,我们提出两项创新:训练时采用批处理秩感知权重估计,推理时使用紧凑记忆库合成难负样本。SRAIN在复合视频检索中表现最佳,在复合图像检索中达到当前最先进水平,同时相比基于MLLM的方法显著降低查询延迟。
原文摘要 · Abstract (English)
At the heart of composed visual data retrieval is the fusion of a reference visual input and a textual modification into a single query. While current state-of-the-art methods utilize multimodal large language models for this fusion, their complexity introduces prohibitive querytime latency, limiting their scalability. We instead revisit the efficacy of simple linear interpolation within an embedding space, and introduce SRAIN, the first framework that dynamically predicts query-specific interpolation weights. The key challenge lies in the fact that the quality of an interpolation weight should be measured by the interpolated embedding's discriminability from negatives as well as its proximity to true targets; this makes collecting and predicting optimal weights intractable. We overcome this bottleneck through two key innovations: batch-wise rank-aware weight estimation during training, and a compact memory bank that synthesizes hard negatives during inference. SRAIN achieves the best in composed video retrieval and matches the current state of the art in composed image retrieval, all while substantially reducing querytime latency compared to MLLM-based alternatives.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。