arXiv:2512.20916cs.IRcs.MM2025-12

用多模态大模型总结物品并检索协同信号,实现高效可解释的推荐。

MMSRARec: Summarization and Retrieval Augumented Sequential Recommendation Based on Multimodal Large Language Model

  • 用大模型生成物品关键词摘要,自适应控制摘要长度与信息保留。
  • 将用户行为中的协同信号转为关键词,作为补充上下文增强推荐。
  • 兼顾性能、可解释性与计算成本,适合追求透明推荐的场景。

多模态大语言模型(MLLM)在推荐系统中展现出巨大潜力,但其在多模态序列推荐中的应用仍不充分。现有方法多利用预训练MLLM的语义理解能力生成物品嵌入或语义标识,但表示可解释性差,难迁移至基于语言模型的推荐系统;另一类方法将用户行为序列转换为图文对进行多轮MLLM推理,计算开销巨大;且多数方法忽视协同信号的融合。为此,本文提出多模态摘要与检索增强的序列推荐框架(MMSRARec)。首先,利用MLLM将物品摘要为精炼关键词,并通过结合摘要长度、信息损失与重建难度的奖励函数微调模型,实现摘要策略的自适应调整;其次,受检索增强生成启发,将协同信号转化为对应关键词并作为补充上下文注入;最后,采用多任务监督微调对齐MLLM与多模态序列推荐任务。在多个主流推荐数据集上的实验表明,MMSRARec能高效、可解释地理解用户行为历史与物品信息,实现精准推荐。

原文摘要 · Abstract (English)

Recent advancements in Multimodal Large Language Models (MLLMs) have demonstrated significant potential in recommendation systems. However, the effective application of MLLMs to multimodal sequential recommendation remains unexplored: A) Existing methods primarily leverage the multimodal semantic understanding capabilities of pre-trained MLLMs to generate item embeddings or semantic IDs, thereby enhancing traditional recommendation models. These approaches generate item representations that exhibit limited interpretability, and pose challenges when transferring to language model-based recommendation systems. B) Other approaches convert user behavior sequence into image-text pairs and perform recommendation through multiple MLLM inference, incurring prohibitive computational and time costs. C) Current MLLM-based recommendation systems generally neglect the integration of collaborative signals. To address these limitations while balancing recommendation performance, interpretability, and computational cost, this paper proposes MultiModal Summarization-and-Retrieval-Augmented Sequential Recommendation. Specifically, we first employ MLLM to summarize items into concise keywords and fine-tune the model using rewards that incorporate summary length, information loss, and reconstruction difficulty, thereby enabling adaptive adjustment of the summarization policy. Inspired by retrieval-augmented generation, we then transform collaborative signals into corresponding keywords and integrate them as supplementary context. Finally, we apply supervised fine-tuning with multi-task learning to align the MLLM with the multimodal sequential recommendation. Extensive evaluations on common recommendation datasets demonstrate the effectiveness of MMSRARec, showcasing its capability to efficiently and interpretably understand user behavior histories and item information for accurate recommendations.

序列推荐多模态大模型可解释

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。