arXiv:2603.12625cs.IRcs.AI2026-03

用大模型将图像转为语义描述,提升推荐准确性

VLM4Rec: Multimodal Semantic Representation for Recommendation with Large Vision-Language Models

  • 用大视觉语言模型将图片转为自然语言描述,实现语义对齐
  • 在多个数据集上超越原始视觉特征和融合方法,提升推荐效果
  • 适合需要精准理解商品语义的电商、内容推荐场景

多模态推荐常被视作特征融合问题,即结合文本与视觉信号以建模用户偏好。然而,其效果不仅取决于模态融合方式,更关键在于物品内容是否在语义空间中与偏好匹配。因为原始视觉特征仅保留外观相似性,而用户决策通常受风格、材质、使用场景等高层语义驱动。为此,我们提出基于大视觉语言模型的多模态语义表示框架VLM4Rec,不依赖直接特征融合,而是通过大模型将每张物品图像映射为显式自然语言描述,并编码为面向偏好的密集项表示。推荐通过历史项嵌入的简单基于用户画像的语义匹配完成,实现高效的离线-在线解耦。在多个多模态推荐数据集上的实验表明,VLM4Rec持续优于原始视觉特征及多种融合基线,表明在该场景下表示质量比融合复杂度更重要。代码已开源:https://github.com/tyvalencia/enhancing-mm-rec-sys。

原文摘要 · Abstract (English)

Multimodal recommendation is commonly framed as a feature fusion problem, where textual and visual signals are combined to better model user preference. However, the effectiveness of multimodal recommendation may depend not only on how modalities are fused, but also on whether item content is represented in a semantic space aligned with preference matching. This issue is particularly important because raw visual features often preserve appearance similarity, while user decisions are typically driven by higher-level semantic factors such as style, material, and usage context. Motivated by this observation, we propose LVLM-grounded Multimodal Semantic Representation for Recommendation (VLM4Rec), a lightweight framework that organizes multimodal item content through semantic alignment rather than direct feature fusion. VLM4Rec first uses a large vision-language model to ground each item image into an explicit natural-language description, and then encodes the grounded semantics into dense item representations for preference-oriented retrieval. Recommendation is subsequently performed through a simple profile-based semantic matching mechanism over historical item embeddings, yielding a practical offline-online decomposition. Extensive experiments on multiple multimodal recommendation datasets show that VLM4Rec consistently improves performance over raw visual features and several fusion-based alternatives, suggesting that representation quality may matter more than fusion complexity in this setting. The code is released at https://github.com/tyvalencia/enhancing-mm-rec-sys.

多模态推荐视觉语言模型语义对齐推荐系统

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。