测试大模型在互动叙事中保持长期逻辑一致性的能力
Can LLM Agents Stick to the Script? A Benchmark for Long-Horizon Consistency in Interactive Narratives
- 构建100个基于电影剧本的叙事环境，自动检测角色承诺是否持续
- 最强模型仅42%能在20轮后不出现逻辑冲突，冲突率高达40%-68%
- 适合研究叙事一致性、游戏AI或长程对话系统的开发者
大型语言模型的快速发展正在推动游戏领域中的开放性互动叙事。然而，现有研究忽视了用户自由干预下维持长期逻辑一致性和叙事完整性的关键挑战。为此，我们提出叙事承诺保持（NCP）问题，并以互动叙事为测试场景。引入NCP-Bench基准，包含100个源自电影梗概的叙事环境，每个环境均具备结构化叙事规范（轨迹、承诺与初始事实），可在玩家代理与叙述者代理交互过程中自动验证。对前沿大模型的实验表明存在显著的长期一致性差距：高语言质量并不保证承诺保持；即使强模型在对抗性干预下也频繁生成逻辑冲突内容，最佳模型GPT-5.2在20轮后存活率仅为42%，各模型的事实冲突率介于40%至68%之间，仅有零星运行满足全部成就承诺且在100轮内完成。
原文摘要 · Abstract (English)
The rapid advancement of Large Language Models (LLMs) is revolutionizing AI for Games by enabling open-ended and fluid interactive storytelling. However, existing research has largely overlooked the critical challenge of maintaining long-horizon logical consistency and narrative integrity against unconstrained user interventions. To address this, we formulate this challenge as Narrative Commitment Preservation (NCP), and take interactive narrative as our testbed. We introduce NCP-Bench, a benchmark of 100 narrative environments derived from movie synopses. Each environment includes a structured narrative specification (trajectory, commitments, and initial facts) that we can automatically check throughout the interaction between the player agent and the narrator agent. Experiments across state-of-the-art LLMs reveal a substantial long-horizon consistency gap: high linguistic quality does not guarantee commitment preservation; even strong models frequently generate logically conflicting content under adversarial interventions, with the best-performing model (GPT-5.2) achieving only 42% survival rate after 20 turns and fact conflict rates ranging from 40% to 68% across models, and only isolated runs satisfying all achievement commitments within the 100-turn limit.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。