让大模型根据问题重要性调整回答深度，提升决策质量。
Every Question Has Its Own Value: Reinforcement Learning with Explicit Human Values
- 直接将人类价值信号融入奖励函数，指导模型优先处理重要任务。
- 在多类任务中，价值加权准确率显著高于仅关注正确性的基线方法。
- 能自动判断问题价值：高价值问题答得更详细，低价值则简洁回应。
我们提出强化学习中的显式人类价值（RLEV）方法，将可量化的价值信号直接引入大语言模型的优化过程。尽管基于可验证奖励的强化学习（RLVR）在客观任务中通过二值正确性奖励有效训练模型，但忽略了任务之间的相对重要性差异。RLEV通过在奖励函数中整合人类定义的价值信号，扩展了该框架。利用带有明确真实价值标签的考试风格数据集，RLEV在多种强化学习算法和模型规模下均优于仅依赖正确性的基线方法。关键的是，RLEV策略不仅提升了价值加权准确率，还学会了一种价值敏感的终止策略：对低价值提示简洁回应，对高价值提示深入作答。我们证明这一行为源于序列末端标记上的价值加权梯度放大机制。消融实验确认该性能提升与价值对齐存在因果关系。即使在价值信号存在噪声（如难度标签）的情况下，RLEV仍保持鲁棒性，表明优化显式效用函数是实现大模型与人类优先级对齐的可行路径。
原文摘要 · Abstract (English)
We propose Reinforcement Learning with Explicit Human Values (RLEV), a method that aligns Large Language Model (LLM) optimization directly with quantifiable human value signals. While Reinforcement Learning with Verifiable Rewards (RLVR) effectively trains models in objective domains using binary correctness rewards, it overlooks that not all tasks are equally significant. RLEV extends this framework by incorporating human-defined value signals directly into the reward function. Using exam-style data with explicit ground-truth value labels, RLEV consistently outperforms correctness-only baselines across multiple RL algorithms and model scales. Crucially, RLEV policies not only improve value-weighted accuracy but also learn a value-sensitive termination policy: concise for low-value prompts, thorough for high-value ones. We demonstrate this behavior stems from value-weighted gradient amplification on end-of-sequence tokens. Ablation studies confirm the gain is causally linked to value alignment. RLEV remains robust under noisy value signals, such as difficulty-based labels, demonstrating that optimizing for an explicit utility function offers a practical path to aligning LLMs with human priorities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。