用大模型生成的多模态嵌入,让推荐系统真正理解图文内容。
Do Recommender Systems Really Leverage Multimodal Content? A Comprehensive Analysis on Multimodal Representations for Recommendation
- 用结构化提示让大视觉语言模型直接生成对齐的图文嵌入
- 相比传统方法,推荐准确率显著提升,且无需复杂融合策略
- 嵌入可转为文本描述,适合需要可解释性的推荐场景
多模态推荐系统通过整合图像和文本等异构内容提升推荐精度。然而,其性能提升是否源于真正的多模态理解,还是仅因模型复杂度增加仍不明确。本文聚焦多模态项目嵌入的语义信息量,发现标准提取器(如ResNet50、Sentence-Bert)虽能提升性能,但依赖模态特定编码器和随意融合策略,缺乏跨模态对齐控制。为此,我们利用大视觉语言模型(LVLMs)通过结构化提示生成原生多模态嵌入,实现无须融合的语义对齐表示。在多个设置下的实验显示性能显著提升。此外,LVLM嵌入可解码为结构化文本描述,将其作为辅助内容引入推荐系统后,进一步改善推荐效果,实证验证了其编码的深层语义与对齐能力。研究强调语义丰富表示的重要性,表明LVLMs是构建可靠多模态推荐表示的有力基础。
原文摘要 · Abstract (English)
Multimodal Recommender Systems aim to improve recommendation accuracy by integrating heterogeneous content, such as images and textual metadata. While effective, it remains unclear whether their gains stem from true multimodal understanding or increased model complexity. This work investigates the role of multimodal item embeddings, emphasizing the semantic informativeness of the representations. Initial experiments reveal that embeddings from standard extractors (e.g., ResNet50, Sentence-Bert) enhance performance, but rely on modality-specific encoders and ad hoc fusion strategies that lack control over cross-modal alignment. To overcome these limitations, we leverage Large Vision-Language Models (LVLMs) to generate multimodal-by-design embeddings via structured prompts. This approach yields semantically aligned representations without requiring any fusion. Experiments across multiple settings show notable performance improvements. Furthermore, LVLMs embeddings offer a distinctive advantage: they can be decoded into structured textual descriptions, enabling direct assessment of their multimodal comprehension. When such descriptions are incorporated as side content into recommender systems, they improve recommendation performance, empirically validating the semantic depth and alignment encoded within LVLMs outputs. Our study highlights the importance of semantically rich representations and positions LVLMs as a compelling foundation for building robust and meaningful multimodal representations in recommendation tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。