用分层强化学习提升对话智能体的策略与回复质量。
ToSCA: Leveraging Hierarchical Reinforcement Learning on Temporal and Strategic Abstractions of Conversational Agents
- 分两层建模：高层定策略，低层生成回复，实现时间与策略抽象融合。
- 在日常与情感支持对话中，策略与回复质量均优于多个基线方法。
- 设计双粒度奖励机制，缓解稀疏奖励问题，提升训练稳定性。
人类在推理与交互中自然表现出多种抽象形式，包括跨决策时序的时间抽象和对沟通意图的战略抽象。受此启发，我们提出一种两级分层强化学习（HRL）框架，用于对话智能体，弥合现有逐标记级与逐语句级强化学习方法之间的差距。该框架基于两级马尔可夫决策过程（MDP），使标记级回复生成依赖于由显式文本策略表示的语句级动作。基于理论分析与效率考量，我们采用DQN优化高层Q网络，使用PPO训练低层演员-评论家模型。为缓解奖励稀疏性并促进收敛，引入双粒度奖励机制，结合语句级满意度得分、标记级内在一致性以及KL散度惩罚。在日常对话与情感支持对话上的实验表明，该方法在策略判断与回复质量上均持续优于多种基线模型。代码已开源：https://github.com/AaronJi/ToSCA。
原文摘要 · Abstract (English)
Humans naturally exhibit multiple forms of abstraction in reasoning and interaction, including temporal abstraction across decision timescales and strategic abstraction over communicative intents. Inspired by these complementary abstractions, we propose a two-level hierarchical reinforcement learning (HRL) framework for conversational agents that bridges the gap between existing token-level and utterance-level RL methods. Built upon a two-level Markov decision process (MDP), our framework conditions token-level response generation on utterance-level actions represented by explicit textual strategies. Based on theoretical analysis and efficiency considerations, we employ DQN to optimize the high-level Q-network and PPO to train the low-level actor-critic. To further alleviate reward sparsity and facilitate convergence, we introduce a dual-granularity reward mechanism that combines the utterance-level satisfaction score with token-level intrinsic self-consistency and a KL-divergence penalty. Experiments on both daily-life and emotional support conversations demonstrate that our method consistently outperforms a wide range of baselines in both strategy determination and response quality. Our implementation is available at https://github.com/AaronJi/ToSCA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。