对比大模型与传统方法，发现人工编码仍更准，大模型需人工验证。
Computational frame analysis revisited: On LLMs for studying news coverage
- 用新构建的金标准数据集对比大模型、小模型和人工编码
- 大模型在新闻框架识别中始终不如人工，部分弱于小型语言模型
- 建议混合使用多种方法，强调研究者应选择合适工具组合
以往计算方法在媒体框架识别中虽有潜力但也存在缺陷。生成式大模型如GPT和Claude正被广泛用于内容分析，但其实际效果如何？我们通过系统评估它们与传统方法（词袋模型、编码器仅模型）及人工编码的差异来回答该问题。研究基于六个月内美国2022年猴痘疫情新闻报道构建了新的金标准数据集，采用归纳迭代方式开发。结果表明，尽管生成式大模型有潜在用途，但整体表现持续落后于人工编码者，某些情况下甚至不及小型语言模型。任何模型选择都需人类验证。通过分析不同任务下各方法的适用性，我们揭示了它们之间的互补性，为研究者提供协同使用策略。最终倡导方法多元并行，并提出未来计算框架分析的发展路线图。
原文摘要 · Abstract (English)
Computational approaches have previously shown various promises and pitfalls when it comes to the reliable identification of media frames. Generative LLMs like GPT and Claude are increasingly being used as content analytical tools, but how effective are they for frame analysis? We address this question by systematically evaluating them against their computational predecessors: bag-of-words models and encoder-only transformers; and traditional manual coding procedures. Our analysis rests on a novel gold standard dataset that we inductively and iteratively developed through the study, investigating six months of news coverage of the US Mpox epidemic of 2022. While we discover some potential applications for generative LLMs, we demonstrate that they were consistently outperformed by manual coders, and in some instances, by smaller language models. Some form of human validation was always necessary to determine appropriate model choice. Additionally, by examining how the suitability of various approaches depended on the nature of different tasks that were part of our frame analytical workflow, we provide insights as to how researchers may leverage the complementarity of these approaches to use them in tandem. We conclude by endorsing a methodologically pluralistic approach and put forth a roadmap for computational frame analysis for researchers going forward.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。