用生成属性提升电商商品细粒度图文表示,显著改善相似商品检索效果。
AFMRL: Attribute-Enhanced Fine-Grained Multi-Modal Representation Learning in E-commerce

- 将商品细粒度理解转化为属性生成任务,利用多模态大模型提取关键属性。
- 两阶段训练:先用属性指导对比学习筛选难样本,再以检索效果反哺属性生成。
- 在多个电商数据集上达到领先性能,适合需要精准商品匹配的场景。
图文表征对电商任务(如完全相同商品检索)至关重要。尽管大型表征模型(如VLM2Vec)具备强大的多模态理解能力,但在区分高度相似商品时仍缺乏细粒度语义理解。为此,我们提出属性增强的细粒度多模态表征学习方法(AFMRL),将商品细粒度理解定义为属性生成任务。该方法利用多模态大语言模型(MLLM)从商品图像和文本中生成关键属性,并通过两阶段训练框架提升表征学习:1)属性引导的对比学习(AGCL),使用MLLM生成的关键属性指导图像-文本对比学习,识别难样本并过滤噪声负样本;2)检索感知的属性强化(RAR),利用表征模型在融合属性后的检索性能作为奖励信号,在多模态微调中增强MLLM的属性生成能力。大规模电商数据集上的实验表明,该方法在多个下游检索任务上达到当前最优性能,验证了利用生成模型推动细粒度表征学习的有效性。
原文摘要 · Abstract (English)
Multimodal representation is crucial for E-commerce tasks such as identical product retrieval. Large representation models (e.g., VLM2Vec) demonstrate strong multimodal understanding capabilities, yet they struggle with fine-grained semantic comprehension, which is essential for distinguishing highly similar items. To address this, we propose Attribute-Enhanced Fine-Grained Multi-Modal Representation Learning (AFMRL), which defines product fine-grained understanding as an attribute generation task. It leverages the generative power of Multimodal Large Language Models (MLLMs) to extract key attributes from product images and text, and enhances representation learning through a two-stage training framework: 1) Attribute-Guided Contrastive Learning (AGCL), where the key attributes generated by the MLLM are used in the image-text contrastive learning training process to identify hard samples and filter out noisy false negatives. 2) Retrieval-aware Attribute Reinforcement (RAR), where the improved retrieval performance of the representation model post-attribute integration serves as a reward signal to enhance MLLM's attribute generation during multimodal fine-tuning. Extensive experiments on large-scale E-commerce datasets demonstrate that our method achieves state-of-the-art performance on multiple downstream retrieval tasks, validating the effectiveness of harnessing generative models to advance fine-grained representation learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。