重新评估编程知识追踪模型,发现性能差异可能源于实验设计问题。
Ensuring Reliability in Programming Knowledge Tracing: A Re-evaluation of Attention-augmented Models and Experimental Protocols
- 统一超参数选择与序列构造标准,避免实验偏差
- 注意力模型性能提升不如原报告显著,复杂度未带来稳定优势
- 适合关注模型可复现性与公平比较的研究者
编程知识追踪(PKT)近年通过融合注意力机制与RNN的混合方法取得进展。尽管这些模型表现优异,但其可靠性易受实现细节和实验设计影响。本研究重新审视代表性PKT模型,发现注意力维度设置不当会扭曲性能评估,且忽略服务器时间戳导致学生作答顺序错乱,破坏时间因果性,造成结果过度乐观。为确保一致评估,采用单一折固定超参数并全交叉验证保持不变。进一步分析作业特征及最大序列长度的影响。在CodeWorkout数据集上重评估后发现,在严格控制条件下,注意力增强模型与标准DKT的性能差距大幅缩小,架构复杂度并未持续带来性能提升。本工作为可靠、可比的编程知识追踪评估提供实用指南。
原文摘要 · Abstract (English)
Programming Knowledge Tracing (PKT) has recently advanced through hybrid approaches that integrate attention-based feature modeling for code representation with RNN-based sequential prediction. While these models report strong empirical performance, their reliability can be sensitive to subtle implementation and experimental design choices. This study revisits representative PKT models and shows that reported gains can be substantially influenced by model configuration and sequence construction practices. We identify issues in attention dimension settings that affect performance estimates, and demonstrate that improper ordering of student attempts, such as ignoring ServerTimestamp, can violate temporal causality and lead to overly optimistic results. To ensure consistent evaluation, hyperparameters are selected via grid search guided by a single designated fold and then fixed uniformly across all folds during cross-validation. We further analyze the role of assignment-wise characteristics and systematically explore the impact of maximum sequence length. Using this protocol, we re-evaluate PKT models on the CodeWorkout dataset. Our results show that, under controlled and consistent settings, the performance gap between attention-enhanced models and standard DKT is significantly reduced, and increased architectural complexity does not consistently translate into superior performance. Beyond individual model comparisons, this work provides practical guidance for reliable and comparable evaluation in programming knowledge tracing.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。