用强化学习让AI学会规划长篇故事，生成更连贯的下一章。
Learning to Reason for Long-Form Story Generation
- 设计新任务与奖励机制，从无标注书籍中学习故事推理能力。
- 生成的章节在人类评估中显著优于基线，尤其在科幻奇幻类更突出。
- 无需人工标注，适合需要长期逻辑一致性的故事生成场景。
生成数千词级别的高质量长篇故事需掌握情节、角色发展及风格一致性等多项能力。由于缺乏标注数据和精确质量评估手段，现有基于大语言模型的方法多依赖手动设计的提示策略来激发类作者行为，过程繁琐且任务依赖性强。受数学与编程领域中可验证奖励强化学习成功的启发，我们提出一个通用的故事生成任务（下一章预测）和奖励形式（通过完成率提升实现验证奖励），利用未标注的书籍数据作为学习信号，使模型能对故事的浓缩信息进行推理并生成详细下一章计划。我们的推理能力通过所生成章节的质量进行评估，并与未训练和监督微调（SFT）基线对比。成对的人类评估显示，由学习推理生成的章节在几乎所有指标上更受青睐，且在科幻与奇幻类型中效果尤为明显。
原文摘要 · Abstract (English)
Generating high-quality stories spanning thousands of tokens requires competency across a variety of skills, from tracking plot and character arcs to keeping a consistent and engaging style. Due to the difficulty of sourcing labeled datasets and precise quality measurements, most work using large language models (LLMs) for long-form story generation uses combinations of hand-designed prompting techniques to elicit author-like behavior. This is a manual process that is highly dependent on the specific story-generation task. Motivated by the recent success of applying RL with Verifiable Rewards to domains like math and coding, we propose a general story-generation task (Next-Chapter Prediction) and a reward formulation (Verified Rewards via Completion Likelihood Improvement) that allows us to use an unlabeled book dataset as a learning signal for reasoning. We learn to reason over a story's condensed information and generate a detailed plan for the next chapter. Our reasoning is evaluated via the chapters it helps a story-generator create, and compared against non-trained and supervised finetuning (SFT) baselines. Pairwise human judgments reveal the chapters our learned reasoning produces are preferred across almost all metrics, and the effect is more pronounced in Scifi and Fantasy genres.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。