arXiv:2504.03906cs.CL2025-04被引 3

构建首个气候议题多模态数据集,评估大模型在社交媒体中的气候回应质量。

CliME: Evaluating Multimodal Climate Discourse on Social Media and the Climate Alignment Quotient (CAQ)

  • 设计多模态数据集与五维评估指标CAQ,系统衡量大模型气候话语能力。
  • 发现多数模型在行动力维度表现差,仅Claude 3.7 Sonnet综合表现最佳。
  • 适合关注气候传播、AI伦理与社会影响的研究者使用。

大型语言模型(LLMs)在理解气候相关语境方面的能力引发关注。尽管气候变化是社交媒体的热点话题,其多模态表达分析仍不充分,现有工具难以判断大模型是否放大可信解决方案或传播未经证实的主张。为此,我们提出CliME(Climate Change Multimodal Evaluation),首个多模态数据集,包含2579条推特与Reddit帖子。该基准涵盖幽默表情包和怀疑性内容,揭示这些形式如何将复杂议题简化为具有影响力的病毒叙事,影响公众认知与政策讨论。为系统评估大模型性能,我们引入气候对齐指数(CAQ),包含五个维度:表达清晰度、证据支持、共鸣度、过渡自然性与具体性。此外,提出三个分析视角:可操作性、批判性与公正性,指导基于CAQ的大模型气候话语评估。基于CAQ的实验结果表明,多数模型在批判性与公正性上表现良好,但在可操作性维度持续表现不佳。在所评估模型中,Claude 3.7 Sonnet综合表现最优。我们公开发布CliME数据集与代码,以推动该领域研究。

原文摘要 · Abstract (English)

The rise of Large Language Models (LLMs) has raised questions about their ability to understand climate-related contexts. Though climate change dominates social media, analyzing its multimodal expressions is understudied, and current tools have failed to determine whether LLMs amplify credible solutions or spread unsubstantiated claims. To address this, we introduce CliME (Climate Change Multimodal Evaluation), a first-of-its-kind multimodal dataset, comprising 2579 Twitter and Reddit posts. The benchmark features a diverse collection of humorous memes and skeptical posts, capturing how these formats distill complex issues into viral narratives that shape public opinion and policy discussions. To systematically evaluate LLM performance, we present the Climate Alignment Quotient (CAQ), a novel metric comprising five distinct dimensions: Articulation, Evidence, Resonance, Transition, and Specificity. Additionally, we propose three analytical lenses: Actionability, Criticality, and Justice, to guide the assessment of LLM-generated climate discourse using CAQ. Our findings, based on the CAQ metric, indicate that while most evaluated LLMs perform relatively well in Criticality and Justice, they consistently underperform on the Actionability axis. Among the models evaluated, Claude 3.7 Sonnet achieves the highest overall performance. We publicly release our CliME dataset and code to foster further research in this domain.

气候传播多模态评估大模型评测社会影响

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。