实验证明图文嵌入未必都有效,文本单独用效果接近全模态。
Are Multimodal Embeddings Truly Beneficial for Recommendation? A Deep Dive into Whole vs. Individual Modalities
- 通过移除图文嵌入测试其实际作用,评估14个主流模型
- 图文联合提升性能,但简单融合方法增益有限;文本单独使用已接近全模态效果
- 图像单独使用效果差,提示应重视文本模态的主导作用
多模态推荐已成为主流范式,通常利用Sentence-BERT、Vision Transformers和ResNet等预训练模型提取的文本与视觉嵌入。这一做法基于直观假设:融合多模态嵌入可提升推荐性能。然而,尽管广泛采用,该假设缺乏全面的实证验证。为此,本文提出核心研究问题:多模态嵌入是否真正有益于推荐?我们开展大规模实证研究,考察文本与视觉嵌入在现代多模态推荐模型中作为整体及单独使用时的作用。具体包括两个关键问题:(1) 多模态嵌入整体是否提升推荐性能?(2) 每个独立模态(文本或图像)单独使用是否有效?为隔离各模态影响,采用模态击穿策略,将对应嵌入设为常数或随机噪声。评估了14个主流先进多模态推荐模型。结果表明:(1) 多模态嵌入总体上提升性能,尤其在图结构融合模型中表现更优;而常用基线模型如VBPR和BM3在简单融合下增益有限;(2) 文本模态单独使用时性能普遍接近全模态设置,而图像模态单独使用则表现不佳。这些发现为多模态推荐研究提供了基础洞见与实践指导。
原文摘要 · Abstract (English)
Multimodal recommendation has emerged as a mainstream paradigm, typically leveraging text and visual embeddings extracted from pre-trained models such as Sentence-BERT, Vision Transformers, and ResNet. This approach is founded on the intuitive assumption that incorporating multimodal embeddings can enhance recommendation performance. However, despite its popularity, this assumption lacks comprehensive empirical verification. This presents a critical research gap. To address it, we pose the central research question of this paper: Are multimodal embeddings truly beneficial for recommendation? To answer this question, we conduct a large-scale empirical study examining the role of text and visual embeddings in modern multimodal recommendation models, both as a whole and individually. Specifically, we pose two key research questions: (1) Do multimodal embeddings as a whole improve recommendation performance? (2) Is each individual modality - text and image - useful when used alone? To isolate the effect of individual modalities - text or visual - we employ a modality knockout strategy by setting the corresponding embeddings to either constant values or random noise. To ensure the scale and comprehensiveness of our study, we evaluate 14 widely used state-of-the-art multimodal recommendation models. Our findings reveal that: (1) multimodal embeddings generally enhance recommendation performance - particularly when integrated through more sophisticated graph-based fusion models. Surprisingly, commonly adopted baseline models with simple fusion schemes, such as VBPR and BM3, show only limited gains. (2) The text modality alone achieves performance comparable to the full multimodal setting in most cases, whereas the image modality alone does not. These results offer foundational insights and practical guidance for the multimodal recommendation community.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。