用好奇心奖励提升大模型输出多样性，同时保持对齐效果。
Curiosity-Driven Reinforcement Learning from Human Feedback
- 引入内在好奇心奖励，激励模型探索新文本状态。
- 在摘要与指令遵循任务中，多样性显著提升，对齐性能不变。
- 适合关注生成多样性与可控性平衡的研究者。
基于人类反馈的强化学习（RLHF）在对齐大型语言模型（LLMs）与人类偏好方面表现优异，但常导致输出多样性下降。这一多样性和对齐质量之间的权衡仍是重大挑战。受强化学习中好奇心驱动探索的启发，我们提出好奇心驱动的RLHF（CD-RLHF），在传统稀疏外在奖励基础上引入对新颖状态的内在奖励，以同时优化输出多样性和对齐质量。我们在文本摘要和指令遵循等任务上进行了广泛实验，结果表明，该方法在多个面向多样性的指标上实现显著提升，同时保持与标准RLHF相当的人类偏好对齐水平。代码已公开于 https://github.com/ernie-research/CD-RLHF。
原文摘要 · Abstract (English)
Reinforcement learning from human feedback (RLHF) has proven effective in aligning large language models (LLMs) with human preferences, but often at the cost of reduced output diversity. This trade-off between diversity and alignment quality remains a significant challenge. Drawing inspiration from curiosity-driven exploration in reinforcement learning, we introduce curiosity-driven RLHF (CD-RLHF), a framework that incorporates intrinsic rewards for novel states, alongside traditional sparse extrinsic rewards, to optimize both output diversity and alignment quality. We demonstrate the effectiveness of CD-RLHF through extensive experiments on a range of tasks, including text summarization and instruction following. Our approach achieves significant gains in diversity on multiple diversity-oriented metrics while maintaining alignment with human preferences comparable to standard RLHF. We make our code publicly available at https://github.com/ernie-research/CD-RLHF.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。