arXiv:2509.26093cs.CL2025-09被引 8

用分层专家网络优化对话推荐系统的交互策略,提升个性化推荐效果。

Reinforced Strategy Optimization for Conversational Recommender Systems via Network-of-Experts

  • 分宏策略规划与微响应生成两层,由不同专家协同决策。
  • 在两个公开数据集上显著优于当前最佳基线,提升对话性能。
  • 适合研究对话系统、推荐算法的学者与工程师参考。

对话推荐系统(CRS)通过多轮自然语言交互为用户提供个性化推荐。鉴于大语言模型(LLMs)强大的交互与推理能力,利用LLM构建CRS成为新趋势。然而,现有方法通常未显式优化交互策略,仅依赖统一提示和模型内知识决定如何互动,导致效果不佳。本文提出一种新的强化策略优化(RSO)方法,通过专家网络架构将策略驱动的响应生成分解为宏观策略规划与微观策略适配。宏观层面,规划者专家选择宏观策略(如推荐、解释、鼓励);微观层面,执行者专家根据选定策略生成详细回应,并由提供用户偏好与事实依据的辅助专家指导。该分层设计解耦了各子任务的优化,使每层学习更高效。针对高质量多轮训练数据稀缺问题,将策略学习建模为强化学习问题,由基于LLM的奖励模型引导自动策略探索。大量实验表明,相较于现有最优基线,RSO显著提升交互性能,验证了显式分层策略优化的有效性。

原文摘要 · Abstract (English)

Conversational Recommender Systems (CRSs) aim to provide personalized recommendations through multi-turn natural language interactions with users. Given the strong interaction and reasoning skills of Large Language Models (LLMs), leveraging LLMs for CRSs has recently emerged as a promising direction. However, existing LLM-based methods often lack explicit optimization of interaction strategies, instead relying on unified prompts and the LLM's internal knowledge to decide how to interact, which can lead to suboptimal outcomes. In this paper, we propose a novel Reinforced Strategy Optimization (RSO) method for CRS, which decomposes the process of generating strategy-driven response decisions into the macro-level strategy planning and micro-level strategy adaptation through a network-of-experts architecture. At the macro level, a Planner expert selects macro-level interaction strategies (e.g., recommend, explain, encourage). At the micro level, an Actor expert generates detailed responses conditioned on the selected macro-level strategy, guided by auxiliary experts that provide complementary information such as user preferences and factual grounding. This hierarchical decomposition disentangles the optimization of different sub-tasks involved in CRS response generation, enabling more tractable learning at each level. To address the scarcity of high-quality multi-turn training data, we formulate strategy learning as a reinforcement learning problem, guided by an LLM-based reward model to achieve automatic strategy exploration. Extensive experiments show that RSO significantly improves interaction performance compared to state-of-the-art baselines, demonstrating the effectiveness of explicit hierarchical strategy optimization for CRS.

对话推荐强化学习专家网络

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。