用AI评估学生写作中的反论,发现其逻辑性体现批判性思维,且AI评分与人评高度一致。
Counterargument for Critical Thinking as Judged by AI and Humans

- 让学生对AI生成内容写反论,检验批判性思维能力。
- 学生自写反论具备逻辑性,符合批判性思维核心要求。
- 六款前沿大模型评分与人类评委高度一致,可规模化应用。
本干预研究探讨在生成式人工智能(GenAI)背景下,学生写作中运用反论对批判性思维的影响。鉴于使用GenAI存在作弊和认知卸载的风险,我们选取某大学36名学生,让他们从一组热门争议话题中任选一个主题撰写文章。采用六项成熟量表(聚焦、逻辑、内容、风格、正确性、引用)对每篇合格稿件(共35篇,剔除1篇异常)进行三次人工评估(两名学生互评+一位经验教师),评分采用五级李克特量表。同时,使用六款前沿大语言模型作为评判者,遵循相同标准进行评估。研究采用混合方法设计,包含定量分析与定性开放反馈。结果表明:(1) 学生针对AI生成内容所写的反论具备逻辑性等关键特征,体现批判性思维;(2) GenAI可基于清晰量表实现大规模评估,其评分与人类评估总体一致,各模型的Gwet AC2组内一致性达0.33(除一款外)。
原文摘要 · Abstract (English)
This intervention study investigates the use of counterarguments in writing for critical thinking by students in the context of Generative AI (GenAI). This is especially as risks of cheating and cognitive offloading exist with the use of GenAI. We presented 36 students in a particular university course with 4 carefully selected thesis statements (from a set of popular debates) to write about anyone of them. We used six established rubrics (focus, logic, content, style, correctness and reference) to conduct three human assessments (two student peer-reviews and one experienced teacher) per writeup on a 5-point Likert scale for all the qualified samples (n) of 35 submissions (after disqualifying one for irregularity). Using the same rubrics and guidelines, we also assessed the submissions using six frontier LLMs as judges. Our mixed-method design included qualitative open-ended feedback per assessment and quantitative methods. The results reveal that (1) the students' self-written counterarguments to AI-generated content contains logic, among other things, which is a key component of critical thinking, and (2) GenAI can be successfully used at scale to assess students' written work, based on clear rubrics, and these assessments generally align with human assessments as shown with Gwets AC2 inter-rater reliability values of 0.33 for all the models except one.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。