arXiv:2605.04831cs.CLcs.AI2026-05被引 2

构建首个故事生成偏好评估基准,提升大模型讲故事的符合人类审美的能力。

StoryAlign: Evaluating and Training Reward Models for Story Generation

论文配图:StoryAlign: Evaluating and Training Reward Models for Story Generation
图 1 · 摘自论文原文
  • 提出StoryRMB基准,含1133组人工验证的故事偏好数据。
  • 新模型StoryReward在该基准上达66.3%准确率,优于更大模型。
  • 适用于需要高质量叙事生成的研究与应用,如内容创作、教育场景。

故事生成旨在自动产出连贯、结构清晰且引人入胜的叙事。尽管大语言模型(LLMs)显著提升了文本生成能力,但其生成的故事在复杂叙事结构和人类偏好契合度上仍落后于人类创作。根本原因在于缺乏对人类故事偏好的有效建模,而这类偏好本身具有主观性且研究不足。本文系统评估了人类故事偏好建模效果,提出首个相关基准StoryRMB,包含1,133个高质量、人工验证的样本,每个样本包含一个提示、一个被选中的故事和三个被拒绝的故事。我们发现现有奖励模型在选择人类偏好的故事时表现不佳,最佳模型准确率仅66.3%。为此,我们在多个领域构建约10万对高质量故事偏好数据,开发出StoryReward,一种基于该数据集训练的先进故事偏好奖励模型。StoryReward在StoryRMB上达到当前最优性能,超越许多更大模型。此外,将其用于下游测试时缩放应用(如Best-of-N),可更优地选出符合人类偏好的故事。相关代码与数据已开源:https://github.com/THU-KEG/StoryReward。

原文摘要 · Abstract (English)

Story generation aims to automatically produce coherent, structured, and engaging narratives. Although large language models (LLMs) have significantly advanced text generation, stories generated by LLMs still diverge from human-authored works regarding complex narrative structure and human-aligned preferences. A key reason is the absence of effective modeling of human story preferences, which are inherently subjective and under-explored. In this work, we systematically evaluate the modeling of human story preferences and introduce StoryRMB, the first benchmark for assessing reward models on story preferences. StoryRMB contains $1,133$ high-quality, human-verified instances, each consisting of a prompt, one chosen story, and three rejected stories. We find existing reward models struggle to select human-preferred stories, with the best model achieving only $66.3\%$ accuracy. To address this limitation, we construct roughly $100,000$ high-quality story preference pairs across diverse domains and develop StoryReward, an advanced reward model for story preference trained on this dataset. StoryReward achieves state-of-the-art (SoTA) performance on StoryRMB, outperforming much larger models. We also adopt StoryReward in downstream test-time scaling applications for best-of-n (BoN) story selection and find that it generally chooses stories better aligned with human preferences. We will release our dataset, model, and code to facilitate future research. Related code and data are available at https://github.com/THU-KEG/StoryReward.

故事生成奖励模型偏好对齐评估基准

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。