用语义感知奖励提升自由生成任务的长文本质量
Semantically-Aware Rewards for Open-Ended R1 Training in Free-Form Generation
- 基于双评价数据集训练评分模型PrefBERT，区分好坏输出
- 在多句到段落级文本上表现稳定，优于传统指标
- 适合需要高质量长文本生成的研究者和开发者
开放域长文本生成的评估极具挑战性，因难以界定优质输出的标准。现有方法常忽略连贯性、风格或相关性，或受预训练数据偏见影响。为此，我们提出PrefBERT，一种用于GRPO框架的评分模型，通过为优劣输出设计差异化奖励来指导训练。该模型在两个包含多样长文本风格与李克特量表评分的数据集上训练，能提供比传统指标ROUGE-L和BERTScore更优的语义奖励反馈。经大模型评判、人工评分及定性分析验证，PrefBERT在多句至段落级文本上仍具可靠性，与GRPO所需的可验证奖励高度对齐。人工评估证实，使用PrefBERT作为奖励信号训练策略模型，生成结果更符合人类偏好。代码已开源。
原文摘要 · Abstract (English)
Evaluating open-ended long-form generation is challenging because it is hard to define what clearly separates good from bad outputs. Existing methods often miss key aspects like coherence, style, or relevance, or are biased by pretraining data, making open-ended long-form evaluation an underexplored problem. To address this gap, we propose PrefBERT, a scoring model for evaluating open-ended long-form generation in GRPO and guiding its training with distinct rewards for good and bad outputs. Trained on two response evaluation datasets with diverse long-form styles and Likert-rated quality, PrefBERT effectively supports GRPO by offering better semantic reward feedback than traditional metrics ROUGE-L and BERTScore do. Through comprehensive evaluations, including LLM-as-a-judge, human ratings, and qualitative analysis, we show that PrefBERT, trained on multi-sentence and paragraph-length responses, remains reliable across varied long passages and aligns well with the verifiable rewards GRPO needs. Human evaluations confirm that using PrefBERT as the reward signal to train policy models yields responses better aligned with human preferences than those trained with traditional metrics. Our code is available at https://github.com/zli12321/long_form_rl.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。