让多模态大模型更会推理,提升推荐系统精准度
MLLMRec-R1: Incentivizing Reasoning Capability in Large Language Models for Multimodal Sequential Recommendation
- 将视觉信息离线转为文本,降低计算开销
- 通过可信推理样本训练,避免奖励虚高
- 适合做多模态推荐的模型优化,尤其关注推理能力
群组相对策略优化(GRPO)已成为提升大语言模型推理与偏好对齐的标准后训练范式,并在基于LLM的推荐系统中表现出色。然而,将GRPO扩展到多模态序列推荐(MSR)面临根本性挑战:首先,MSR需联合编码历史交互与候选项的视觉内容,导致视觉标记主导输入,且群组推演成本随历史长度和候选集规模线性增长,使GRPO训练代价过高;其次,现有思维链(CoT)监督在推荐场景中存在奖励膨胀问题,更高训练奖励未必带来排名性能提升,可能诱发捷径学习。为此,我们提出MLLMRec-R1,一种高效稳定的多模态序列推荐GRPO框架。该方法离线将视觉信号文本化,消除昂贵的视觉标记同时保留多模态语义;通过精炼与置信度感知评估构建高质量多模态CoT监督;并采用混合粒度数据增强策略,选择性注入可靠CoT样本,保留标准训练数据,缓解奖励膨胀,提升泛化稳定性。在三个基准数据集上的大量实验表明,MLLMRec-R1持续优于当前最优方法,建立了可行有效的多模态序列推荐GRPO推理管道。代码已公开于https://github.com/wangyu0627/MLLMRec-R1。
原文摘要 · Abstract (English)
Group relative policy optimization (GRPO) has become a standard post-training paradigm for improving reasoning and preference alignment in large language models (LLMs), and has recently shown strong effectiveness in LLM-based recommender systems. However, extending GRPO-based reasoning pipelines to multimodal sequential recommendation (MSR) with multimodal large language models (MLLMs) faces fundamental obstacles. First, MSR requires jointly encoding visual content for both historical interactions and multiple candidate items, causing visual tokens to dominate the input and making the cost of group-based rollout scale with history length and candidate set size, which renders GRPO-based training prohibitively expensive. Second, existing Chain-of-Thought (CoT) supervision suffers from reward inflation in recommendation scenarios, where higher training rewards do not reliably translate into improved ranking performance and may induce shortcut learning. To address these challenges, we propose MLLMRec-R1, an efficient and stable GRPO-based reasoning framework for multimodal sequential recommendation. MLLMRec-R1 textualizes visual signals offline to eliminate expensive visual tokens while preserving multimodal semantics, and constructs high-quality multimodal CoT supervision through refinement and confidence-aware assessment. Furthermore, a mixed-grained data augmentation strategy selectively injects reliable CoT samples while retaining standard training data, mitigating reward inflation and improving generalization stability. Extensive experiments on three benchmark datasets demonstrate that MLLMRec-R1 consistently outperforms state-of-the-art methods, establishing a practical and effective GRPO-based reasoning pipeline for multimodal sequential recommendation. The code is available at https://github.com/wangyu0627/MLLMRec-R1.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。