用图结构标注通胀叙事,降低人为解读差异。
From Variance to Invariance: Qualitative Content Analysis for Narrative Graph Annotation
- 将叙事转为有向无环图,节点表事件,边表因果
- 局部约束表示可降低标注变异性,重叠度量会高估一致性
- 适合做叙事分析与高质量标注的NLP研究者
新闻话语中的叙事在塑造公众对经济事件(如通货膨胀)的理解中起关键作用。对这些叙事进行结构化标注与评估仍是自然语言处理的关键挑战。本文提出一种融合定性内容分析原则的叙事图标注框架,通过减少标注误差来提升标注质量。我们构建了一个通货膨胀叙事数据集,以有向无环图(DAG)形式标注,其中节点代表事件,边编码因果关系。为评估标注质量,采用6×3因子实验设计,考察叙事表示方式(六种)和距离度量类型(三种)对标注者间一致性的影(克里彭多夫α)。分析显示:(1) 宽松度量(基于重叠的距离)会高估可靠性;(2) 局部约束表示(如一跳邻居)能有效降低标注变异性。标注与图版克里彭多夫α实现已开源。该框架与评估结果为应对人类标签变异(HLV)的图结构叙事标注提供了实用指导。
原文摘要 · Abstract (English)
Narratives in news discourse play a critical role in shaping public understanding of economic events, such as inflation. Annotating and evaluating these narratives in a structured manner remains a key challenge for Natural Language Processing (NLP). In this work, we introduce a narrative graph annotation framework that integrates principles from qualitative content analysis (QCA) to prioritize annotation quality by reducing annotation errors. We present a dataset of inflation narratives annotated as directed acyclic graphs (DAGs), where nodes represent events and edges encode causal relations. To evaluate annotation quality, we employed a $6\times3$ factorial experimental design to examine the effects of narrative representation (six levels) and distance metric type (three levels) on inter-annotator agreement (Krippendorrf's $α$), capturing the presence of human label variation (HLV) in narrative interpretations. Our analysis shows that (1) lenient metrics (overlap-based distance) overestimate reliability, and (2) locally-constrained representations (e.g., one-hop neighbors) reduce annotation variability. Our annotation and implementation of graph-based Krippendorrf's $α$ are open-sourced. The annotation framework and evaluation results provide practical guidance for NLP research on graph-based narrative annotation under HLV.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。