用视觉语言模型和大模型分析学生行为,考虑同伴互动提升课堂参与度识别准确率
Context Matters: Peer-Aware Student Behavioral Engagement Measurement via VLM Action Parsing and LLM Sequence Classification
- 用少样本微调的视觉语言模型识别学生动作类别
- 2分钟视频分段处理,生成动作序列,提升对连续行为的捕捉能力
- 结合同伴行为上下文,通过大模型判断学生是否专注,适合教育智能研究者
理解课堂中的学生行为对于提升教学质量和学生参与度至关重要。现有方法预测学生参与度通常需要大量标注数据来建模行为多样性,但隐私问题常限制研究者仅使用自有私有数据集。此外,同龄人行为所体现的课堂上下文常被忽略。为解决上述问题,我们提出一种基于视频的学生参与度测量三阶段新框架。首先,探索视觉-语言模型在少样本情况下的适应性,微调以区分不同动作类别,仅需少量训练样本。其次,为应对持续且不可预测的学生行为,采用滑动时间窗技术将每位学生的2分钟视频划分为非重叠片段,每个片段由微调后的VLM模型分配动作类别,生成动作序列。最后,利用大语言模型对整个动作序列及课堂上下文进行分类,判断学生属于专注或分心状态。实验结果表明该方法在识别学生参与度方面具有有效性。源代码将公开于https://github.com/ahmed-nady/context_aware_student_engagement。
原文摘要 · Abstract (English)
Understanding student behavior in the classroom is essential to improve both pedagogical quality and student engagement. Existing methods for predicting student engagement typically require substantial annotated data to model the diversity of student behaviors, yet privacy concerns often restrict researchers to their own proprietary datasets. Moreover, the classroom context, represented in peers' actions, is ignored. To address the aforementioned limitation, we propose a novel three-stage framework for video-based student engagement measurement. First, we explore the few-shot adaptation of the vision-language model for student action recognition, which is fine-tuned to distinguish among action categories with a few training samples. Second, to handle continuous and unpredictable student actions, we utilize the sliding temporal window technique to divide each student's 2-minute-long video into non-overlapping segments. Each segment is assigned an action category via the fine-tuned VLM model, generating a sequence of action predictions. Finally, we leverage the large language model to classify this entire sequence of actions, together with the classroom context, as belonging to an engaged or disengaged student. The experimental results demonstrate the effectiveness of the proposed approach in identifying student engagement. The source code will be available at https://github.com/ahmed-nady/context_aware_student_engagement.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。