用ChatGPT自动编码协作问题解决中的沟通数据，提升评估效率。
Automated Coding of Communications in Collaborative Problem-solving Tasks Using ChatGPT
- 用ChatGPT对五组数据的沟通内容进行自动编码，适配两种框架。
- 新推理模型如GPT-o1-mini未显著提升编码准确率，效果因任务而异。
- 根据误编码案例优化提示词可部分提升精度，但不适用于所有任务。
协作问题解决（CPS）被广泛认为是21世纪关键能力。其评估高度依赖基于构念相关框架对沟通数据进行人工编码，这一过程长期成为规模化评估的瓶颈。基于五个数据集和两种编码框架，我们证明了ChatGPT可在合理水平上完成沟通数据编码，但性能随不同ChatGPT模型、编码框架及任务特性而异。有趣的是，更侧重推理的新模型如GPT-o1-mini和GPT-o3-mini并未带来更高编码表现。此外，通过分析误编码案例优化提示词可在某些任务中提升准确率，但该方法效果并不一致。这些发现为研究者和实践者开发可扩展、高效的沟通数据分析方法提供了实用指导。
原文摘要 · Abstract (English)
Collaborative problem solving (CPS) is widely recognized as a critical 21st-century skill. Assessing CPS depends heavily on coding the communication data using a construct-relevant framework, and this process has long been a major bottleneck to scaling up such assessments. Based on five datasets and two coding frameworks, we demonstrate that ChatGPT can code communication data to a satisfactory level, though performance varies across ChatGPT models, and depends on the coding framework and task characteristics. Interestingly, newer reasoning-focused models such as GPT-o1-mini and GPT-o3-mini do not necessarily yield better coding results. Additionally, we show that refining prompts based on feedback from miscoded cases can improve coding accuracy in some instances, though the effectiveness of this approach is not consistent across all tasks. These findings offer practical guidance for researchers and practitioners in developing scalable, efficient methods to analyze communication data in support of 21st-century skill assessment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。