arXiv:2512.07777cs.CL2025-12Conference of the …被引 1

大模型能识别故事不连贯,但判断结果却不一致,说明理解仍有缺陷。

Mary, the Cheeseburger-Eating Vegetarian: Do LLMs Recognize Incoherence in Narratives?

  • 用成对故事数据测试模型内部表征对不连贯性的识别能力
  • 模型在不同提示下无法稳定区分连贯与不连贯故事
  • 更关注场景违例而非角色特质矛盾,依赖常识而非逻辑推理

基于成对叙事数据集,我们研究了大语言模型(LLMs)在可靠区分连贯与不连贯故事方面的能力。探针实验表明,模型内部表征能有效识别不连贯叙事。然而,在回答评分问题时,模型在多种提示变体下未能满意地区分两类故事,暗示其对叙事理解存在差距。测试的推理模型也未消除这一缺陷,表明思维链可能无法弥合模型内部状态与行为之间的差异。此外,模型对违反设定的情节(如沙漠下雨)更敏感,而非角色违背既定特质(如素食者点汉堡),表明其更依赖典型世界知识而非基于意义的叙事一致性。结果中发现的一致不对称性表明,模型并未完全掌握叙事连贯性。

原文摘要 · Abstract (English)

Leveraging a dataset of paired narratives, we investigate the extent to which large language models (LLMs) can reliably separate incoherent and coherent stories. A probing study finds that LLMs' internal representations can reliably identify incoherent narratives. However, LLMs generate responses to rating questions that fail to satisfactorily separate the coherent and incoherent narratives across several prompt variations, hinting at a gap in LLM's understanding of storytelling. The reasoning LLMs tested do not eliminate these deficits, indicating that thought strings may not be able to fully address the discrepancy between model internal state and behavior. Additionally, we find that LLMs appear to be more sensitive to incoherence resulting from an event that violates the setting (e.g., a rainy day in the desert) than to incoherence arising from a character violating an established trait (e.g., Mary, a vegetarian, later orders a cheeseburger), suggesting that LLMs may rely more on prototypical world knowledge than building meaning-based narrative coherence. The consistent asymmetry found in our results suggests that LLMs do not have a complete grasp on narrative coherence.

大模型叙事理解连贯性认知偏差

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。