用改进的ICAP框架量化协作对话中的认知参与度，比较人工与大模型标注效果。
Measuring Cognitive Engagement in Collaborative Discourse with an Extended ICAP Framework: Comparing Human Annotation, In-Context Learning, and Reflective LLM Agents
- 基于7点ICAP框架分析协作对话中的认知参与水平。
- 人工标注一致性高（kappa=0.906-0.998），大模型标注较低（kappa=0.541-0.609）。
- 反思型大模型能提升标注一致性，适合教育研究与智能辅导系统开发。
协作促进学习与问题解决，但其成效依赖于对话过程中的认知参与度。本研究采用扩展的7点ICAP框架（包含互动、建构、主动、被动四种模式）来刻画协作对话中认知参与的差异。参与度由受训人工标注员编码，并与基于大语言模型（LLM）的标注方法进行比较，包括上下文学习（ICL）、零样本提示和自反思代理。人工标注者间的一致性在框架迭代阶段保持稳健（kappa = 0.906–0.998），显著高于基于ICL的标注（kappa = 0.541–0.609）。经人工优化的框架使人工标注一致性提升（Delta kappa = 0.10），但对基于ICL的LLM仅带来小于0.04的微弱改善。代理优化框架虽提升了跨模型一致性，但仍低于人工优化框架。研究结果凸显了代理方法的潜力，强调未来需持续结合理论指导的人工标注与基于大模型的方法。
原文摘要 · Abstract (English)
Collaboration supports learning and problem-solving, but its effectiveness depends on cognitive engagement during discourse. This study applies an extended 7-point ICAP framework based on the Interactive, Constructive, Active, and Passive modes to characterize variation in cognitive engagement during collaborative dialogue. Engagement was coded by trained human annotators and compared with large language model (LLM)-based labeling approaches, including in-context learning (ICL), zero-shot prompting, and self-reflective agents. Interrater reliability among human annotators was robust across framework refinement stages (kappa = 0.906-0.998), higher than the moderate agreement observed for ICL-based annotation (kappa = 0.541-0.609). The human-refined framework improved agreement among human annotators (Delta kappa = 0.10), but produced only modest gains for ICL-based LLMs (Delta kappa less than 0.04). Agent-refined frameworks improved cross-model agreement but remained below the human-refined framework. These findings highlight the promise of agent-based approaches and the importance of continued interaction between theory-guided human annotation and LLM-based methods in future work.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。