用模拟用户反馈优化多轮推荐,让大模型更懂用户真实偏好。
User Simulator-Guided Multi-Turn Preference Optimization for Reasoning LLM-based Conversational Recommendation
- 通过多任务微调让模拟用户更贴近真实反馈
- 结合强化学习与精细奖励设计,提升推荐准确率
- 适合研究多轮对话推荐与大模型个性化交互的学者
对话式推荐系统(CRS)依赖自然语言交互实现个性化推荐,但信息稀疏的对话历史和单轮推荐范式严重限制了复杂用户偏好的建模。近期研究引入基于大模型的用户模拟器,生成自然语言反馈并进行多轮模拟交互以辅助推荐。然而,由于模拟器在推理阶段无法获取真实用户偏好标签,其反馈可能偏离实际兴趣,导致错误累积,严重影响推荐器泛化能力。受大模型多步推理能力和强化学习在策略优化中的有效性启发,我们提出 SMTPO 框架——一种用户模拟器引导的多轮偏好优化对话推荐方法。为在无显式标签条件下对齐模拟反馈与真实偏好,我们通过多任务监督微调(SFT)提升反馈质量,使模拟器更精准反映用户复杂多样需求。针对偏差反馈导致多轮优化不稳定的问题,先通过 SFT 让基于推理的大模型学习偏好推理与推荐模式,再采用细粒度奖励设计的强化学习逐步对齐真实用户偏好,显著提升推荐性能。在多个公开数据集上的大量实验验证了该方法的有效性与可迁移性。
原文摘要 · Abstract (English)
Conversational Recommender Systems (CRSs) leverage natural language interactions for personalized recommendation, yet information-scarce dialogue histories and single-turn recommendation paradigms may severely hinder accurate modeling of complex user preferences. To alleviate this issue, recent studies have introduced LLM-based user simulators, which generate natural language feedback and perform simulated multi-turn interactions to assist recommendation. Nevertheless, since simulators cannot access true user preference labels during inference, their feedback may deviate from actual user interests, causing errors to accumulate over multiple interactions and severely affecting the generalization of the recommender. Inspired by the multi-step reasoning capabilities of LLMs and the effectiveness of reinforcement learning in policy optimization, we propose SMTPO, a user simulator-guided multi-turn preference optimization conversational recommendation framework. To align simulator-generated feedback with true user preferences in the absence of explicit labels, we enhance feedback quality via multi-task supervised fine-tuning (SFT), enabling the simulator to better reflect users' complex and diverse needs. To address the challenge of biased feedback destabilizing multi-turn optimization, we first allow the reasoning LLM-based recommender to learn preference reasoning and recommendation patterns through SFT and then employ reinforcement learning with fine-grained reward design to progressively align with true user preferences, improving recommendation performance. Extensive experiments on public datasets demonstrate the effectiveness and transferability of our method.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。