用AI反馈提升长文本大模型表现，效果显著且不冲突。
LongReward: Improving Long-context Large Language Models with AI Feedback
- 用现成大模型从四个维度评估长文本回复质量
- 结合离线强化学习使长文本模型性能大幅提升
- 既提升长文本能力，也保持短指令遵循性
尽管长上下文大语言模型（LLMs）已取得显著进展，但用于监督微调（SFT）的合成数据质量下降，制约了模型在长文本场景下的表现。理论上，引入合适的奖励信号可进一步增强模型能力，但如何在长上下文场景中获取可靠奖励仍缺乏研究。为此，我们提出LongReward，一种利用现成大模型从四个符合人类价值的维度——帮助性、逻辑性、忠实性和完整性——对长文本模型输出进行评分的方法，并设计了对应的评估流程。通过结合LongReward与离线强化学习算法DPO，我们有效提升了长文本SFT模型的表现。实验表明，LongReward不仅显著改善了模型在长上下文任务中的表现，还增强了其对短指令的遵循能力。此外，长上下文DPO与传统短上下文DPO可并行使用，互不影响性能。
原文摘要 · Abstract (English)
Though significant advancements have been achieved in developing long-context large language models (LLMs), the compromised quality of LLM-synthesized data for supervised fine-tuning (SFT) often affects the long-context performance of SFT models and leads to inherent limitations. In principle, reinforcement learning (RL) with appropriate reward signals can further enhance models' capacities. However, how to obtain reliable rewards in long-context scenarios remains unexplored. To this end, we propose LongReward, a novel method that utilizes an off-the-shelf LLM to provide rewards for long-context model responses from four human-valued dimensions: helpfulness, logicality, faithfulness, and completeness, each with a carefully designed assessment pipeline. By combining LongReward and offline RL algorithm DPO, we are able to effectively improve long-context SFT models. Our experiments indicate that LongReward not only significantly improves models' long-context performance but also enhances their ability to follow short instructions. We also find that long-context DPO with LongReward and conventional short-context DPO can be used together without hurting either one's performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。