用可验证情绪奖励训练对话模型,提升其共情能力。
RLVER: Reinforcement Learning with Verifiable Emotion Rewards for Empathetic Agents
- 构建模拟用户生成确定性情绪评分,作为强化学习奖励信号。
- 微调后模型共情基准得分从13.3升至79.2,代码能力基本保留。
- 适合想提升对话智能与情感理解的AI研发人员。
大型语言模型(LLMs)在逻辑和算法推理方面表现优异,但情感智能(EQ)远落后于其认知能力。尽管可验证奖励强化学习(RLVR)已在其他领域取得进展,但在对话场景——尤其是情感智能方面的应用仍不充分。本文提出首个端到端的强化学习框架RLVER,利用模拟用户的可验证情绪奖励来培养LLM的高阶共情能力。在该框架中,自我一致的情感模拟用户参与对话轮次,并在对话中生成确定性情绪分数,作为引导模型学习的奖励信号。对公开的Qwen2.5-7B-Instruct模型使用PPO进行微调后,其Sentient-Benchmark得分从13.3提升至79.2,同时保持了较强的数学与编程能力。大量实验表明:(i) RLVER持续提升多项对话能力;(ii) 思考型与非思考型模型呈现不同趋势——思考型模型在共情与洞察上更优,非思考型则偏重行动;(iii) GRPO通常带来稳定提升,而PPO可将某些能力推向更高上限;(iv) 更具挑战性的环境并非总是更优,适度难度环境反而可能产生更强效果。结果表明,RLVER是实现情感智能与通用能力兼备语言代理的可行路径。
原文摘要 · Abstract (English)
Large language models (LLMs) excel at logical and algorithmic reasoning, yet their emotional intelligence (EQ) still lags far behind their cognitive prowess. While reinforcement learning from verifiable rewards (RLVR) has advanced in other domains, its application to dialogue-especially for emotional intelligence-remains underexplored. In this work, we introduce RLVER, the first end-to-end reinforcement learning framework that leverages verifiable emotion rewards from simulated users to cultivate higher-order empathetic abilities in LLMs. Within this framework, self-consistent affective simulated users engage in dialogue rollouts and produce deterministic emotion scores during conversations, serving as reward signals to guide the LLM's learning. Fine-tuning publicly available Qwen2.5-7B-Instruct model with PPO boosts its Sentient-Benchmark score from 13.3 to 79.2 while largely preserving mathematical and coding competence. Extensive experiments reveal that: (i) RLVER consistently improves multiple dialogue capabilities; (ii) Thinking and non-thinking models show distinct trends--thinking models excel in empathy and insight, while non-thinking models favor action; (iii) GRPO often yields stable gains, while PPO can push certain capabilities to a higher ceiling; (iv) More challenging environments are not always better-moderate ones can yield stronger outcomes. Our results show that RLVER is a practical route toward emotionally intelligent and broadly capable language agents.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。