arXiv:2602.11158cs.HCcs.AI2026-02

比较不同方法对师生AI认知的测量差异,发现结果因分析方式而异。

Methodological Variation in Studying Staff and Student Perceptions of AI

  • 用情感分析与主题分析对比评论和焦点小组数据
  • 同源数据经不同分析得不同结论,如正负倾向与内容主题不一致
  • 提醒研究者注意方法选择对教育AI评估的影响

本文比较了评估师生对人工智能认知的不同方法,探讨这些方法带来的测量差异。研究聚焦于AI感知,通常通过单一定量、定性或混合方法评估。我们收集两类质性数据:独立评论与结构化焦点小组,并对每类数据进行双重分析:情感与立场分析以衡量整体正负面情绪;词云与主题分析则揭示内容细节,特别是学生与教师之间的异同。结果显示,同一数据源经不同分析可得截然不同的结论——整体情绪评分与具体主题分析之间存在显著差异。该现象源于所测构念不同:总体情绪反映与内容深度分析呈现不同图景。研究讨论了其在高校实践中的意义,以及对以往研究比较结果的启示。

原文摘要 · Abstract (English)

In this paper, we compare methodological approaches for comparing student and staff perceptions, and ask: how much do these measures vary across different approaches? We focus on the case of AI perceptions, which are generally assessed via a single quantitative or qualitative measure, or with a mixed methods approach that compares two distinct data sources - e.g. a quantitative questionnaire with qualitative comments. To compare different approaches, we collect two forms of qualitative data: standalone comments and structured focus groups. We conduct two analyses for each data source: with a sentiment and stance analysis, we measure overall negativity/positivity of the comments and focus group conversations, respectively. Meanwhile, word clouds from the comments and a thematic analysis of the focus groups provide further detail on the content of this qualitative data - particularly the thematic analysis, which includes both similarities and differences between students and staff. We show that different analyses can produce different results - for a single data source. This variation stems from the construct being evaluated - an overall measure of positivity/negativity can produce a different picture from more detailed content-based analyses. We discuss the implications of this variation for institutional contexts, and for the comparisons from previous studies.

AI认知质性研究教育评估

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。