用Transformer分析视频特征,自动识别在线学习时的学生专注度。
Transformer-Driven Modeling of Variable Frequency Features for Classifying Student Engagement in Online Learning
- 设计多视角并行Transformer架构,融合视频中的面部与动作信息
- 在5个数据集上准确率最高达99.16%,尤其在疲劳检测上表现突出
- 适合教育AI、智能教学系统开发者参考,可实时反馈学生状态
新冠疫情与互联网普及推动了在线学习的发展,但教师难以实时监控学生参与度。本文提出EngageFormer,一种基于Transformer的视频模态学生参与度分类模型。该模型从输入视频中提取三个视角,通过并行Transformer编码器处理,再经全局编码器整合特征,最后由多层感知机预测参与水平。研究基于开源数据集构建了以学习为中心的情绪状态数据集。在DAiSEE、BAUM-1、YawDD、UTA-RLDD及自建数据集上,准确率分别为63.9%、56.73%、99.16%、65.67%和74.89%。在BAUM-1、DAiSEE和YawDD上达到当前最优性能,证明模型在情绪状态识别上的有效性。在二分类的UTA-RLDD上结果可作为后续研究基准。
原文摘要 · Abstract (English)
The COVID-19 pandemic and the internet's availability have recently boosted online learning. However, monitoring engagement in online learning is a difficult task for teachers. In this context, timely automatic student engagement classification can help teachers in making adaptive adjustments to meet students' needs. This paper proposes EngageFormer, a transformer based architecture with sequence pooling using video modality for engagement classification. The proposed architecture computes three views from the input video and processes them in parallel using transformer encoders; the global encoder then processes the representation from each encoder, and finally, multi layer perceptron (MLP) predicts the engagement level. A learning centered affective state dataset is curated from existing open source databases. The proposed method achieved an accuracy of 63.9%, 56.73%, 99.16%, 65.67%, and 74.89% on Dataset for Affective States in E-Environments (DAiSEE), Bahcesehir University Multimodal Affective Database-1 (BAUM-1), Yawning Detection Dataset (YawDD), University of Texas at Arlington Real-Life Drowsiness Dataset (UTA-RLDD), and curated learning-centered affective state dataset respectively. The achieved results on the BAUM-1, DAiSEE, and YawDD datasets demonstrate state-of-the-art performance, indicating the superiority of the proposed model in accurately classifying affective states on these datasets. Additionally, the results obtained on the UTA-RLDD dataset, which involves two-class classification, serve as a baseline for future research. These results provide a foundation for further investigations and serve as a point of reference for future works to compare and improve upon.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。