提出LEAP协议，让学习系统早期预警更可信。
When Can We Trust Early Warnings? Leakage-Excluded Early Outcome Prediction from LMS Interaction Logs
- 用截断优先策略防止时间泄漏，确保预测只用真实可用数据
- 第3周前后性能明显提升，随机森林最早期表现最佳
- 适合做学习预警研究或评估模型可靠性的学者参考
基于学习管理系统（LMS）日志的早期预警模型旨在尽早预测课程结果以提供及时支持。然而，现有“早期”性能常因时间泄漏被夸大——即使用了预测时尚未可得的信息。本文在时间可用性约束下形式化基于截止点的早期预测，并提出LEAP（泄漏排除早期可用性协议），通过截断优先处理、特征溯源审计，防止截止后信息进入评估。我们在公开的开放大学学习分析数据集（OULAD）上构建多阶段协议，在每周截止点进行受控评估。采用多种标准学习方法，评估指标包括ROC-AUC、PR-AUC、Brier分数和[email protected]。结果显示，随着观察窗口扩大，性能持续提升，第3周左右出现显著增长；随机森林在最早截止点表现最优，梯度提升则后续占优。泄漏消融实验进一步表明，尤其是通过测评信息的时间违规会严重夸大早期性能表现。
原文摘要 · Abstract (English)
Early-warning models built from Learning Management System (LMS) logs aim to predict end-of-course outcomes early enough to enable timely learner support. However, reported "early" performance is often inflated by temporal leakage. This occurs when the pipeline uses information that would not yet be available at the time of prediction. We formalize cutoff-based early outcome prediction under a temporal availability constraint and introduce LEAP (Leakage-Excluded Early-Availability Protocol), which enforces cutoff-first truncation prior to joins and aggregation and audits feature provenance to prevent post-cutoff evidence from entering the benchmark. We instantiate LEAP on the public Open University Learning Analytics Dataset (OULAD) as a multi-step protocol for leakage-controlled evaluation across weekly cutoffs. Using several standard learning methods, we evaluate performance using ROC-AUC, PR-AUC, Brier score, and [email protected]. Results show improving performance as the observation window expands, with a marked gain around week~3; Random Forest performs best at the earliest cutoffs, while Gradient Boosting dominates thereafter. Leakage ablations further show that temporal violations, especially through assessment information, can inflate apparent "early" performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。