arXiv:2410.15182cs.CYcs.CL2024-10EMNLP被引 4

用AI检测网络讨论中的理性谦逊,助力健康公共对话

The Computational Anatomy of Humility: Modeling Intellectual Humility in Online Public Discourse

  • 构建宗教话题评论的谦逊标注体系,训练大模型自动识别
  • 模型宏F1达0.64,优于随机基线但低于人类专家水平
  • 为社交媒体素养研究提供可量化的分析工具

个体在分歧中建设性互动的能力是健康多元社会的关键。在线讨论平台同样如此。现有研究多聚焦于防止极化与假信息传播,但提升对话质量还需培育基本美德。本文聚焦于“理性谦逊”(IH),即承认自身信念可能存在局限。我们手动标注并验证了350条来自Reddit的宗教话题帖子,构建了IH标注手册,并基于此开发了大语言模型(LLM)自动化测量方法。最佳模型在标签层面的宏F1得分为0.64(粗粒度分类为0.70),高于预期随机基线(0.51,粗粒度0.32),但仍低于人类标注者上界(0.85,粗粒度0.83)。结果既揭示了在线理性谦逊识别的难度,也为计算社会科学提供了分析和促进理性谦逊的新基础。

原文摘要 · Abstract (English)

The ability for individuals to constructively engage with one another across lines of difference is a critical feature of a healthy pluralistic society. This is also true in online discussion spaces like social media platforms. To date, much social media research has focused on preventing ills -- like political polarization and the spread of misinformation. While this is important, enhancing the quality of online public discourse requires not just reducing ills but also promoting foundational human virtues. In this study, we focus on one particular virtue: ``intellectual humility'' (IH), or acknowledging the potential limitations in one's own beliefs. Specifically, we explore the development of computational methods for measuring IH at scale. We manually curate and validate an IH codebook on 350 posts about religion drawn from subreddits and use them to develop LLM-based models for automating this measurement. Our best model achieves a Macro-F1 score of 0.64 across labels (and 0.70 when predicting IH/IA/Neutral at the coarse level), higher than an expected naive baseline of 0.51 (0.32 for IH/IA/Neutral) but lower than a human annotator-informed upper bound of 0.85 (0.83 for IH/IA/Neutral). Our results both highlight the challenging nature of detecting IH online -- opening the door to new directions in NLP research -- and also lay a foundation for computational social science researchers interested in analyzing and fostering more IH in online public discourse.

理性谦逊NLP社会计算文本分析

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。