用上下文模型替代词典匹配,更准确识别多语言网络抱怨文本
From a Word-Level Dictionary to Sentence-Level Semantics: Multilingual Grievance Labelling with Contextual Models
- 用上下文理解模型替代关键词匹配,提升语义解析能力
- 在五种语言中,负向文本准确率从0.14提升至0.20,尤其改善隐含与引述表达
- 构建非循环基准测试,避免自洽偏差,结果更可信
抱怨是暴力威胁评估中的重要预警信号。当前常通过词级词典(如Grievance Dictionary)对在线文本进行大规模测量,通过加权词匹配实现快速透明的评分。但此类方法无法判断词语是被主张、引用、否定或谴责。且词典常在自身检索出的样本池上评估,导致高分部分反映的是其自身选择规则的一致性。我们在一个包含2,000条、涵盖五种语言的评估集上发现:词典标记为“随机”的样本几乎全部为词典负例,使得词典宏观AUROC从0.686坍缩至0.500的构造下限。我们保留词典的22个构念分类体系,但以上下文阅读模型替代词项匹配,并在非循环基准上评估——该基准将文本划分为无条件随机、词典正例、词典负例三类。整体阅读完整帖子而非仅目标句,在词典沉默区域效果最佳,使词典负例文本平均精确率从0.14提升至0.20,尤其在引述、隐含及跨句抱怨中增益最大。结果表明,结合上下文理解可更真实地测量抱怨,而使用未被词典选中的文本测试则更具公信力。代码与数据集已开源:https://github.com/behavioral-ds/multilingual_grievance。
原文摘要 · Abstract (English)
Grievance is one of the warning signs analysts look for when assessing threats of violence. It is increasingly measured at scale from online text, most often with word-level lexicons like the Grievance Dictionary that score by matching weighted terms. Such matching is a fast and transparent proxy, but it cannot resolve whether a term is asserted, quoted, negated, or condemned. These lexicons are also often evaluated on pools enriched with the very examples they retrieve, so a high score partly reflects agreement with the lexicon's own selection rule. Examining a five-language, 2{,}000-item evaluation pool, we find its halves separated almost perfectly by the lexicon itself: every item labeled ``random'' is in fact lexicon-negative, so the lexicon's apparent macro-AUROC of 0.686 collapses to a 0.500 floor fixed by construction. We keep the dictionary's 22-construct ontology but replace term matching with context-reading models, evaluated on a non-circular benchmark that separates unconditional-random, lexicon-positive, and lexicon-negative strata across five languages. Reading the full post rather than the target sentence alone helps most where the lexicon is silent, raising average precision on lexicon-negative text from 0.14 to 0.20, with the largest gains on quoted, implicit, and cross-sentence grievance. Together, these results show that grievance is measured more faithfully by reading the surrounding context, and more honestly when tested on text the lexicon did not select. We release our code and benchmark at https://github.com/behavioral-ds/multilingual_grievance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。