不依赖人工数据，用强化学习让大模型自动生成超长高质量文本
LongWriter-Zero: Mastering Ultra-Long Text Generation via Reinforcement Learning
- 从零开始用强化学习训练，无需合成数据或标注
- 在WritingBench和Arena-Write上超越100B级大模型
- 适合需要长文本生成的学术写作、内容创作场景
大语言模型在生成超长文本时面临最大长度限制和质量下降问题。以往方法如LongWriter依赖合成数据进行监督微调，但这类数据构建困难、缺乏连贯性且结构单调。本文提出LongWriter-Zero，完全从基础模型出发，不依赖任何标注或合成数据，采用强化学习引导模型在写作中进行规划与优化。通过专门设计的奖励模型，提升长度控制、文本质量和结构格式。实验表明，基于Qwen2.5-32B训练的LongWriter-Zero，在WritingBench和Arena-Write上全面优于传统SFT方法，性能超越DeepSeek R1和Qwen3-235B等100B级以上模型，已开源数据与模型权重。
原文摘要 · Abstract (English)
Ultra-long generation by large language models (LLMs) is a widely demanded scenario, yet it remains a significant challenge due to their maximum generation length limit and overall quality degradation as sequence length increases. Previous approaches, exemplified by LongWriter, typically rely on ''teaching'', which involves supervised fine-tuning (SFT) on synthetic long-form outputs. However, this strategy heavily depends on synthetic SFT data, which is difficult and costly to construct, often lacks coherence and consistency, and tends to be overly artificial and structurally monotonous. In this work, we propose an incentivization-based approach that, starting entirely from scratch and without relying on any annotated or synthetic data, leverages reinforcement learning (RL) to foster the emergence of ultra-long, high-quality text generation capabilities in LLMs. We perform RL training starting from a base model, similar to R1-Zero, guiding it to engage in reasoning that facilitates planning and refinement during the writing process. To support this, we employ specialized reward models that steer the LLM towards improved length control, writing quality, and structural formatting. Experimental evaluations show that our LongWriter-Zero model, trained from Qwen2.5-32B, consistently outperforms traditional SFT methods on long-form writing tasks, achieving state-of-the-art results across all metrics on WritingBench and Arena-Write, and even surpassing 100B+ models such as DeepSeek R1 and Qwen3-235B. We open-source our data and model checkpoints under https://huggingface.co/THU-KEG/LongWriter-Zero-32B
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。