用视觉语言模型提升多模态推荐，解决模态失衡问题
VLM2Rec: Resolving Modality Collapse in Vision-Language Model Embedders for Multimodal Sequential Recommendation
- 引入弱模态惩罚对比学习，平衡图像与文本优化
- 跨模态拓扑正则化保持多模态几何一致性
- 在多个数据集上超越主流基线，提升推荐精度与鲁棒性
多模态序列推荐通常依赖小型冻结预训练编码器，限制语义表达能力，并阻碍协同过滤（CF）信号充分融入物品表征。受大语言模型作为高容量嵌入器成功的启发，我们探索将视觉语言模型（VLM）作为具备CF感知能力的多模态嵌入器用于序列推荐。然而，我们发现标准对比监督微调（SFT）在适配VLM生成嵌入并注入CF信号时，会加剧固有的模态不平衡：优化过程被某一模态主导，另一模态性能下降，最终损害推荐准确率。为此，我们提出VLM2Rec，一种基于VLM嵌入器的多模态序列推荐框架，旨在促进模态间均衡利用。具体地，我们引入弱模态惩罚对比学习以缓解优化过程中的梯度不平衡，并设计跨模态关系拓扑正则化以保持模态间的几何一致性。实验表明，VLM2Rec在多种场景下均持续优于强基线，在准确率与鲁棒性上表现更优。
原文摘要 · Abstract (English)
Sequential Recommendation (SR) in multimodal settings typically relies on small frozen pretrained encoders, which limits semantic capacity and prevents Collaborative Filtering (CF) signals from being fully integrated into item representations. Inspired by the recent success of Large Language Models (LLMs) as high-capacity embedders, we investigate the use of Vision-Language Models (VLMs) as CF-aware multimodal embedders for SR. However, we find that standard contrastive Supervised Fine-Tuning (SFT), used to adapt VLMs for embedding generation and inject CF signals, can amplify inherent modality imbalance: optimization becomes dominated by one modality while the other degrades, ultimately undermining recommendation accuracy. To address this, we propose VLM2Rec, a VLM embedder-based framework for multimodal sequential recommendation designed to promote balanced modality utilization. Specifically, we introduce Weak-modality Penalized Contrastive Learning to mitigate gradient imbalance during optimization and cross-modal Relational Topology Regularization to preserve geometric consistency between modalities. Experiments show that VLM2Rec consistently improves over strong baselines in both accuracy and robustness across diverse scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。