用AI生成的问答测验，效果堪比人工命题。
Evaluating LLM-Generated Q&A Test: a Student-Centered Study
- 基于GPT-4o-mini自动构建自然语言处理课程测验
- 心理测量学分析显示题目区分度强、难度适中
- 学生与专家评分高，适合教育机构批量开发测评
本研究构建了一套基于AI聊天机器人自动生成可靠问答测验的自动化流程。我们以GPT-4o-mini为基础，为一门自然语言处理课程生成了问答测试，并通过学生和专家对测验的心理测量特性与感知质量进行评估。混合格式的项目反应理论（IRT）分析表明，生成题目的区分度强且难度适宜；学生与专家的星级评分反映出整体质量较高。统一的差异项目功能（DIF）检测识别出两个需审查的题目。结果表明，大模型生成的评估内容在心理测量表现和用户满意度上可媲美人工命题，展示出一种可扩展的AI辅助测评开发路径。
原文摘要 · Abstract (English)
This research prepares an automatic pipeline for generating reliable question-answer (Q&A) tests using AI chatbots. We automatically generated a GPT-4o-mini-based Q&A test for a Natural Language Processing course and evaluated its psychometric and perceived-quality metrics with students and experts. A mixed-format IRT analysis showed that the generated items exhibit strong discrimination and appropriate difficulty, while student and expert star ratings reflect high overall quality. A uniform DIF check identified two items for review. These findings demonstrate that LLM-generated assessments can match human-authored tests in psychometric performance and user satisfaction, illustrating a scalable approach to AI-assisted assessment development.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。