arXiv:2507.17080cs.IRcs.AI2025-07中稿 · RecSys 2025

用视觉定位和大模型增强图文嵌入,提升电商推荐精准度。

VL-CLIP: Enhancing Multimodal Recommendations via Visual Grounding and LLM-Augmented CLIP Embeddings

  • 通过视觉定位细化图像特征,捕捉商品细粒度属性。
  • 用大模型改写商品描述,解决文本模糊问题,提升跨模态匹配。
  • 在百万级商品上验证,点击率提升18.6%,成交额增长4.0%。

多模态学习在当今电商平台中至关重要,能实现精准推荐与商品理解。然而,现有视觉语言模型(如CLIP)在电商推荐系统中面临三大挑战:1)弱物体级对齐,全局图像嵌入无法捕捉细粒度商品属性,导致检索性能不佳;2)文本表征模糊,商品描述常缺乏上下文清晰度,影响跨模态匹配;3)领域不匹配,通用视觉语言模型难以适配电商特定数据。为此,我们提出VL-CLIP框架,通过引入视觉定位实现细粒度视觉理解,并利用大模型代理生成增强的文本嵌入。视觉定位通过定位关键商品来优化图像表示,大模型代理则通过消歧商品描述来增强文本特征。该方法在美国家规模最大的电商平台之一上,对数千万商品进行测试,显著提升检索准确率、多模态检索效果与推荐质量,点击率(CTR)提升18.6%,平均交易成本(ATC)降低15.5%,商品总价值(GMV)增长4.0%。实验表明,该框架在精度与语义对齐上优于CLIP、FashionCLIP和GCL等模型,验证了结合物体感知视觉定位与大模型增强文本表示在鲁棒多模态推荐中的潜力。

原文摘要 · Abstract (English)

Multimodal learning plays a critical role in e-commerce recommendation platforms today, enabling accurate recommendations and product understanding. However, existing vision-language models, such as CLIP, face key challenges in e-commerce recommendation systems: 1) Weak object-level alignment, where global image embeddings fail to capture fine-grained product attributes, leading to suboptimal retrieval performance; 2) Ambiguous textual representations, where product descriptions often lack contextual clarity, affecting cross-modal matching; and 3) Domain mismatch, as generic vision-language models may not generalize well to e-commerce-specific data. To address these limitations, we propose a framework, VL-CLIP, that enhances CLIP embeddings by integrating Visual Grounding for fine-grained visual understanding and an LLM-based agent for generating enriched text embeddings. Visual Grounding refines image representations by localizing key products, while the LLM agent enhances textual features by disambiguating product descriptions. Our approach significantly improves retrieval accuracy, multimodal retrieval effectiveness, and recommendation quality across tens of millions of items on one of the largest e-commerce platforms in the U.S., increasing CTR by 18.6%, ATC by 15.5%, and GMV by 4.0%. Additional experimental results show that our framework outperforms vision-language models, including CLIP, FashionCLIP, and GCL, in both precision and semantic alignment, demonstrating the potential of combining object-aware visual grounding and LLM-enhanced text representation for robust multimodal recommendations.

多模态推荐视觉定位大模型增强电商应用

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。