无需标注即可实现跨模态检索,利用编辑距离弱监督提升效率。
Multimodal Information Retrieval for Open World with Edit Distance Weak Supervision
- 通过样本间编辑距离构建弱监督信号,避免传统标注成本。
- 在缺失人员案例中表现媲美现有系统,支持精确与近似匹配。
- 适合数据稀缺场景,可直接复用现有属性标识,无需微调。
现有多媒体检索模型要么依赖模态特定表示模型构建公共子空间,要么需在模态间进行模式映射以度量相似性。本文目标是避免将检索视为有监督分类任务带来的标注开销,并重用大语言模型和视觉任务中的预训练编码器。提出FemmIR框架,通过示例实现多模态查询的相关结果检索,无需任何相似性标签。该能力对真实世界应用至关重要,因数据标注稀疏且需在不微调的通用框架下实现良好性能。我们构建了新数据集MuQNOL用于该任务的基准测试。方法基于编辑距离引入的弱监督:图编辑距离可调整为考虑样本替换的成本,相关性通过对象间编辑代价的隐式信号衡量。与度量学习或编码网络不同,FemmIR重用高层属性,保持属性值与关系约束,并通过多层级交互得分来计算样本与用户提供的查询示例之间的相关性。在缺失人员案例上使用MuQNOL进行实证评估,FemmIR在提供按需检索结果方面表现优异,既支持精确匹配也支持近似匹配,且仅使用系统中现有的属性标识。
原文摘要 · Abstract (English)
Existing multi-media retrieval models either rely on creating a common subspace with modality-specific representation models or require schema mapping among modalities to measure similarities among multi-media data. Our goal is to avoid the annotation overhead incurred from considering retrieval as a supervised classification task and re-use the pretrained encoders in large language models and vision tasks. We propose "FemmIR", a framework to retrieve multimodal results relevant to information needs expressed with multimodal queries by example without any similarity label. Such identification is necessary for real-world applications where data annotations are scarce and satisfactory performance is required without fine-tuning with a common framework across applications. We curate a new dataset called MuQNOL for benchmarking progress on this task. Our technique is based on weak supervision introduced through edit distance between samples: graph edit distance can be modified to consider the cost of replacing a data sample in terms of its properties, and relevance can be measured through the implicit signal from the amount of edit cost among the objects. Unlike metric learning or encoding networks, FemmIR re-uses the high-level properties and maintains the property value and relationship constraints with a multi-level interaction score between data samples and the query example provided by the user. We empirically evaluate FemmIR on a missing person use case with MuQNOL. FemmIR performs comparably to similar retrieval systems in delivering on-demand retrieval results with exact and approximate similarities while using the existing property identifiers in the system.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。