用大模型助手分析学生课后互动,发现助教能填补时间空白但提问引导不足。
Investigating Student Interaction Patterns with Large Language Model-Powered Course Assistants in Computer Science Courses
- 部署大模型助教至6门计算机课程,服务2000名学生
- 晚间使用率高,初学者互动更频繁,多数回复正确有用
- 生成的高阶思考问题少,适合教育者优化提示词与教学设计
在多数高校,学生难以获得课后及时的学术支持。大型语言模型(LLMs)有望弥补这一缺口,但其与学生的互动通常未被教师监控。我们开发并研究了一个跨多门计算机科学课程部署的LLM驱动课程助教系统,以分析实际使用情况并探讨教学意义。截至2024年春季,该系统已覆盖三所院校的六门课程,服务约2000名学生。交互数据分析显示,系统在晚间和夜间使用强度高,且在入门课程中使用更频繁,表明其有效缓解了时间上的支持缺口并满足新手学习者需求。我们对每门课程抽样200次对话进行人工标注:大多数回复被评定为正确且有帮助,少数存在错误或无用;极少包含专门示例。此外,我们考察了一种基于探究的学习策略:仅约11%的对话包含大模型生成的追问,且高级课程学生常忽略这些提问。基于布卢姆分类法的分析表明,当前大模型在生成高阶认知问题方面能力有限。这些模式提示,应设计更具教学导向的基于大模型的教育系统,并加强教师对提示词、内容与政策的配置参与。
原文摘要 · Abstract (English)
Providing students with flexible and timely academic support is a challenge at most colleges and universities, leaving many students without help outside scheduled hours. Large language models (LLMs) are promising for bridging this gap, but interactions between students and LLMs are rarely overseen by educators. We developed and studied an LLM-powered course assistant deployed across multiple computer science courses to characterize real-world use and understand pedagogical implications. By Spring 2024, our system had been deployed to approximately 2,000 students across six courses at three institutions. Analysis of the interaction data shows that usage remains strong in the evenings and nights and is higher in introductory courses, indicating that our system helps address temporal support gaps and novice learner needs. We sampled 200 conversations per course for manual annotation: most sampled responses were judged correct and helpful, with a small share unhelpful or erroneous; few responses included dedicated examples. We also examined an inquiry-based learning strategy: only around 11% of sampled conversations contained LLM-generated follow-up questions, which were often ignored by students in advanced courses. A Bloom's taxonomy analysis reveals that current LLM capabilities are limited in generating higher-order cognitive questions. These patterns suggest opportunities for pedagogically oriented LLM-based educational systems and greater educator involvement in configuring prompts, content, and policies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。