AI辅助质性编码，人机协作更高效发现网络对话深层含义
Processes Matter: How ML/GAI Approaches Could Support Open Qualitative Coding of Online Discourse Datasets
- 用五种ML/GAI方法与四名人类编码员对比分析聊天数据
- AI擅长识别内容标签，人类更懂对话互动逻辑
- 建议将AI作为并行合作者，依研究过程灵活整合
开放式编码是质性研究中关键的归纳步骤，用于从人类数据中发现并构建概念。然而，在大规模话语数据集上捕捉丰富且细微的‘编码时刻’仍具挑战性。尽管已有研究探索机器学习（ML）/生成式人工智能（GAI）在开放编码中的潜力，但缺乏系统评估。本研究比较了五种近期发布的ML/GAI方法与四位人类编码员在移动学习软件相关在线聊天消息数据集上的编码结果。系统分析揭示了各类方法的优劣，展现出人与AI之间的互补潜力：逐行处理的AI方法能有效识别基于内容的编码，而人类在理解对话动态方面更具优势。我们讨论了嵌入式分析过程如何影响ML/GAI结果。研究建议不应以AI取代人类进行开放编码，而应根据研究者的分析流程，将AI作为并行合作者进行整合。
原文摘要 · Abstract (English)
Open coding, a key inductive step in qualitative research, discovers and constructs concepts from human datasets. However, capturing extensive and nuanced aspects or "coding moments" can be challenging, especially with large discourse datasets. While some studies explore machine learning (ML)/Generative AI (GAI)'s potential for open coding, few evaluation studies exist. We compare open coding results by five recently published ML/GAI approaches and four human coders, using a dataset of online chat messages around a mobile learning software. Our systematic analysis reveals ML/GAI approaches' strengths and weaknesses, uncovering the complementary potential between humans and AI. Line-by-line AI approaches effectively identify content-based codes, while humans excel in interpreting conversational dynamics. We discussed how embedded analytical processes could shape the results of ML/GAI approaches. Instead of replacing humans in open coding, researchers should integrate AI with and according to their analytical processes, e.g., as parallel co-coders.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。