解决服装电商多视角图像检索难题,实现按商品整体特征精准查找。
FashionMV: Product-Level Composed Image Retrieval with Multi-View Fashion Data
- 构建多视角商品级检索框架,支持从多个角度描述匹配目标商品。
- 在127K商品、472K图像的数据集上,模型性能超基线10倍。
- 适合做电商视觉搜索、跨视角商品匹配的开发者与研究者参考。
组合图像检索(CIR)通过参考图像与修改文本检索目标图像。现有方法和数据集均局限于图像级——单张参考图加文本描述,输出单张目标图,但真实电商用户会基于商品多视角进行推理。本文提出视图不完整问题,并定义新的多视角商品级CIR任务。为此构建FashionMV,首个大规模多视角时尚数据集,包含127,000个商品、472,000张多视角图像及超过220,000个CIR三元组,全部由自动化流程结合大模型生成。提出ProCIR框架,基于多模态大语言模型,融合两阶段对话、基于描述对齐与思维链引导三种机制,并可选监督微调注入结构化商品知识。在三个时尚基准上的16种配置消融实验表明:(1) 对齐是关键机制;(2) 两阶段对话架构是有效对齐的前提;(3) 微调与思维链存在部分冗余的知识注入路径。最优0.8B参数模型超越所有基线,包括比其大10倍的通用嵌入模型。数据集、模型与代码已开源。
原文摘要 · Abstract (English)
Composed Image Retrieval (CIR) retrieves target images using a reference image paired with modification text. Despite rapid advances, all existing methods and datasets operate at the image level -- a single reference image plus modification text in, a single target image out -- while real e-commerce users reason about products shown from multiple viewpoints. We term this mismatch View Incompleteness and formally define a new Multi-View CIR task that generalizes standard CIR from image-level to product-level retrieval. To support this task, we construct FashionMV, the first large-scale multi-view fashion dataset for product-level CIR, comprising 127K products, 472K multi-view images, and over 220K CIR triplets, built through a fully automated pipeline leveraging large multimodal models. We further propose ProCIR (Product-level Composed Image Retrieval), a modeling framework built upon a multimodal large language model that employs three complementary mechanisms -- two-stage dialogue, caption-based alignment, and chain-of-thought guidance -- together with an optional supervised fine-tuning (SFT) stage that injects structured product knowledge prior to contrastive training. Systematic ablation across 16 configurations on three fashion benchmarks reveals that: (1) alignment is the single most critical mechanism; (2) the two-stage dialogue architecture is a prerequisite for effective alignment; and (3) SFT and chain-of-thought serve as partially redundant knowledge injection paths. Our best 0.8B-parameter model outperforms all baselines, including general-purpose embedding models 10x its size. The dataset, model, and code are publicly available at https://github.com/yuandaxia2001/FashionMV.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。