构建两个叙事场景中角色位置数据集，测试大模型空间理解能力。
Locations of Characters in Narratives: Andersen and Persuasion Datasets
- 人工标注童话与小说中角色的位置信息，构建两个新数据集。
- 最佳模型在两个数据集上的准确率分别为61.85%和56.06%。
- 适合研究叙事理解、常识推理与大模型阅读能力的学者参考。
机器理解叙事文本中的空间关系是阅读理解的重要课题。为评估AI对角色与位置关系的理解能力，我们构建了两个新数据集：Andersen与Persuasion。Andersen数据集选自安徒生童话15个儿童故事，手动标注角色及其位置；Persuasion数据集则来自简·奥斯汀小说《劝导》的角色位置标注。我们用这些数据集提示大型语言模型（LLMs），通过抽取文本片段并提出角色位置问题进行测试。五款模型中，表现最佳者在Andersen数据集上准确识别位置的比例为61.85%，在Persuasion数据集上为56.06%。
原文摘要 · Abstract (English)
The ability of machines to grasp spatial understanding within narrative contexts is an intriguing aspect of reading comprehension that continues to be studied. Motivated by the goal to test the AI's competence in understanding the relationship between characters and their respective locations in narratives, we introduce two new datasets: Andersen and Persuasion. For the Andersen dataset, we selected fifteen children's stories from "Andersen's Fairy Tales" by Hans Christian Andersen and manually annotated the characters and their respective locations throughout each story. Similarly, for the Persuasion dataset, characters and their locations in the novel "Persuasion" by Jane Austen were also manually annotated. We used these datasets to prompt Large Language Models (LLMs). The prompts are created by extracting excerpts from the stories or the novel and combining them with a question asking the location of a character mentioned in that excerpt. Out of the five LLMs we tested, the best-performing one for the Andersen dataset accurately identified the location in 61.85% of the examples, while for the Persuasion dataset, the best-performing one did so in 56.06% of the cases.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。