用生成式模型预测代码回答，动态选题提升编程能力评估精度
CodeGENCAT: Generative Computerized Adaptive Testing for Open-ended Coding Problems
- 基于学生知识水平生成代码响应，指导题目选择
- 早期测试阶段AUC最高提升4.32%，优于所有基线方法
- 适合编程教育评估、自适应学习系统开发者参考
现有计算机自适应测试（CAT）框架通常根据学生答对概率选择题目，忽视了编程等开放性作答中蕴含的丰富知识信息。本文提出生成式自适应测试框架CodeGENCAT，通过预测学生代码响应来选题。首先构建生成式项目反应理论（GIRT）模型，利用监督微调和直接偏好优化训练，实现知识与代码响应的对齐；其次设计三种题目选择算法，分别衡量不确定性、编码风格多样性及预测代码中的信息量。在两个真实编程教育数据集上的实验表明，CodeGENCAT优于所有基准方法，在测试初期最高提升4.32%的AUC值。
原文摘要 · Abstract (English)
Existing Computerized Adaptive Testing (CAT) frameworks typically select questions based on the predicted likelihood that the student will answer correctly. This design ignores information contained in students' open-ended responses, especially in domains such as programming education, where code structures and bugs contain rich information on student knowledge. In this work, we propose \textbf{Code} \textbf{GEN}erative \textbf{CAT} (\textbf{CodeGENCAT}), a generative CAT framework that selects questions using predicted student code responses. First, we develop a Generative Item Response Theory (GIRT) model that generates code responses conditioned on estimated student knowledge, trained with supervised fine-tuning followed by direct preference optimization for knowledge-response alignment. Second, we introduce three question-selection algorithms that measure uncertainty, coding style diversity, and information from predicted student code responses. Experiments on two real-world programming education datasets show that CodeGENCAT outperforms all CAT baselines, achieving an AUC improvement of up to 4.32\% over the strongest baseline in the early stages of adaptive testing.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。