用语义自适应分段，提升大模型对齐中的奖励分配精度
Adaptive Segment-level Reward: Bridging the Gap Between Action and Reward Space in Alignment
- 基于语义而非标点自动划分文本段落
- 在对抗样本上成功率达10%提升，多基准测试增1.3%
- 适合需要精准奖励分配的强化学习对齐场景
强化学习在对齐大语言模型与人类偏好方面表现优异。传统方法基于整个序列的总体奖励进行优化，导致信用分配不准确，难以判断应强化或抑制哪些标记。为此，已有逐步和逐标记的方法被提出，但逐步方法依赖标点分割，仍无法精确识别关键标记；而逐标记方法过于细粒度，关注大量无关标记，引入大量噪声。为更准确地为不同标记分配奖励，改善信用分配，我们提出“自适应段级奖励”方法。该方法采用语义而非标点来自适应地划分段落。实验表明，该方法可集成到多种训练流程中。相比未使用本方法的训练方式，本方法在对抗样本上的成功率提升10%，在MMLU、GSM8K、HumanEval等评估基准上均取得1.3%的提升。
原文摘要 · Abstract (English)
Reinforcement Learning (RL) has proven highly effective in aligning Large Language Models (LLMs) with human preferences. Typical RL methods optimize under an overall sequence reward, which can lead to a suboptimal learning process. This reflects a key credit assignment problem: identifying which tokens to reinforce or suppress. To rectify these shortcomings, step-wise and token-wise methods have been proposed. However, step-wise methods rely on punctuation segmentation and still cannot accurately identify the key tokens. The token-level approach is too fine-grained, attending to many unimportant tokens and thus introducing a large amount of noise. To assign more accurate rewards to different tokens, improving credit assignment, we propose the "Adaptive Segment-wise Reward" method. We employ semantic meaning, rather than punctuation, to adaptively delineate segments. Experiments demonstrate that our method can be integrated into various training methods. Compared to training methods \textit{without} our approach, our method improves the success rate on adversarial samples by 10\%, and achieves a 1.3\% improvement on evaluation benchmarks such as MMLU, GSM8K, HumanEval, etc.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。