通过分步反馈与外部评审，提升大模型长文本生成的准确性和完整性。
LongDPO: Unlock Better Long-form Generation Abilities for LLMs via Critique-augmented Stepwise Information
- 用蒙特卡洛树搜索生成分步偏好对，结合全局记忆保持连贯性。
- 引入外部评审优化候选内容，使长文本生成更符合需求，长度偏差降低。
- 适用于学术写作和代码生成场景，适合追求高质量长文本的用户。
长文本生成在学术论文撰写和仓库级代码生成中至关重要。尽管如此，当前模型（包括GPT-4o）仍表现不佳。现有基于结果监督的偏好学习方法难以提供长上下文的详细反馈，导致内容不完整、长度偏差和质量下降。本文提出通过过程监督增强长文本生成能力。我们采用蒙特卡洛树搜索获取分步偏好对，并利用全局记忆池维持一致性。为解决候选选择不佳的问题，引入外部评审以优化偏好对质量。最后，使用分步DPO训练模型。实验表明，该方法在长文本生成基准上显著提升长度和质量，同时在各类模型主干上对通用任务几乎无损。
原文摘要 · Abstract (English)
Long-form generation is crucial for academic writing papers and repo-level code generation. Despite this, current models, including GPT-4o, still exhibit unsatisfactory performance. Existing methods that utilize preference learning with outcome supervision often fail to provide detailed feedback for extended contexts. This shortcoming can lead to content that does not fully satisfy query requirements, resulting in issues like length deviations, and diminished quality. In this paper, we propose enhancing long-form generation by incorporating process supervision. We employ Monte Carlo Tree Search to gather stepwise preference pairs, utilizing a global memory pool to maintain consistency. To address the issue of suboptimal candidate selection, we integrate external critiques to refine and improve the quality of the preference pairs. Finally, we apply step-level DPO using the collected stepwise preference pairs. Experimental results show that our method improves length and quality on long-form generation benchmarks, with almost lossless performance on general benchmarks across various model backbones.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。