通过随机实验验证了引导学生用AI当学习伙伴比当答案提供者更有效。
Transforming GenAI Policy to Prompting Instruction: An RCT of Scalable Prompting Interventions in a CS1 Course
- 基于ICAP框架设计四类渐进式提示训练,提升学生与AI互动的认知参与度。
- 所有干预组提示技能均提升,参与越深进步越大,且即时测试表现预测期末成绩。
- 方案可推广至不同教学场景,为生成式AI时代的教育政策落地提供实证支持。
尽管生成式AI已普遍使用,学生仍难以区分任务完成与真实学习,缺乏利用AI促进学习的能力,导致无反思使用AI时考试表现更差。然而,尚无大规模随机对照试验(RCT)验证以提示作为导师而非答案提供者的教学干预效果。为此,我们开展了一项为期一学期的随机对照试验(N=979),设置四种基于ICAP框架的教学条件,包含前测、即时后测、延迟后测及问卷调查。混合方法分析显示:(1)所有条件均显著提升提示能力,成效随条件强度从1到4逐步提高,验证了ICAP认知参与层次理论;(2)在前测分数相近的学生中,即时后测学习增益越高,期末考试成绩越好,但组间差异未达显著水平;(3)本干预方案适用于多样化的教育情境、资源和学习者。研究兼具理论与实践意义:(1)理论上,提供了首个大规模RCT,揭示认知参与如何影响提示素养学习,并厘清学习导向提示技能与整体学业表现的关系;(2)实证上,为将生成式AI课堂政策转化为可扩展、可操作的提示素养教学提供了及时的设计指导。
原文摘要 · Abstract (English)
Despite universal GenAI adoption, students cannot distinguish task performance from actual learning and lack skills to leverage AI for learning, leading to worse exam performance when AI use remains unreflective. Yet few interventions teaching students to prompt AI as a tutor rather than solution provider have been validated at scale through randomized controlled trials (RCTs). To bridge this gap, we conducted a semester-long RCT (N=979) with four ICAP framework-based instructional conditions varying in engagement intensity with a pre-test, immediate and delayed post-test and surveys. Mixed methods analysis results showed: (1) All conditions significantly improved prompting skills, with gains increasing progressively from Condition 1 to Condition 4, validating ICAP's cognitive engagement hierarchy; (2) for students with similar pre-test scores, higher learning gain in immediate post-test predict higher final exam score, though no direct between-group differences emerged; (3) Our interventions are suitable and scalable solutions for diverse educational contexts, resources and learners. Together, this study makes empirical and theoretical contributions: (1) theoretically, we provided one of the first large-scale RCTs examining how cognitive engagement shapes learning in prompting literacy and clarifying the relationship between learning-oriented prompting skills and broader academic performance; (2) empirically, we offered timely design guidance for transforming GenAI classroom policies into scalable, actionable prompting literacy instruction to advance learning in the era of Generative AI.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。