真实课堂验证了智能助教跨课程的适应能力与局限。
Learning Engagement Assistant (LEA): Cross-Course Scalability and Classroom Evaluation of an Agentic AI Tutoring System

- 用RAG+知识组件模型构建可交互的智能助教
- 跨三门课程测试,答案相关性稳定在0.88以上
- 生成内容忠实度随课程差异下降,需进一步研究
本文扩展了2026年ICART会议发表的工作,首次在真实课堂中部署学习参与助手LEA(Learning Engagement Assistant),针对8名学生(n=8)进行实际教学实验,并首次评估其在跨课程场景下的可扩展性。系统覆盖三门课程,涵盖两个学术层级和两个学科领域。研究发现,仿真预测与真实部署结果存在差异,仅靠合成学生无法全面反映实际表现。基于RAGAS的跨课程评估(660个问题)显示:答案相关性和上下文精确性在各课程间保持稳定(0.88–0.94 和 0.88–0.90),但生成内容的忠实度随课程与原训练课程的距离增加而下降(0.69至0.50),可能源于生成逻辑对初始课程的过度适配,而非系统本身不可扩展。结果表明,编排层无需调整,但下游组件实现真正课程无关仍需深入探索。
原文摘要 · Abstract (English)
This paper is an extension of a paper presented at the ICAART 2026 conference, which introduced LEA (Learning Engagement Assistant), an adaptive AI tutoring agent combining course-specific Retrieval-Augmented Generation (RAG) with structured Knowledge Component (KC) models across integrated Chat, Tutor, and Quiz modes. That prior work validated LEA on a single STEM course (CMP511) exclusively through simulation, using synthetic learner agents. This paper extends that work by reporting the first classroom deployment of LEA with real students (n = 8, CMP511) and the first empirical test of its cross-course scalability, deploying the system across three courses spanning two academic levels and two disciplinary domains. The study reveals a divergence from simulation predictions across modes, showing that synthetic evaluation alone cannot anticipate all aspects of real deployment. A RAGAS-based cross-course scalability evaluation (660 questions) finds Answer Relevancy and Context Precision broadly stable across courses (0.88-0.94 and 0.88-0.90 respectively), while Faithfulness declines with curriculum distance from the system's original course (0.69 to 0.50), a preliminary finding that may reflect generation logic tuned to the system's original subject rather than a scalability limitation. These findings suggest that while the orchestration layer requires no modification, full course-agnosticism of all downstream components requires further investigation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。