用大模型自动评罗马尼亚高考题，发现评分不稳、算错分等难题。
BacPrep: Lessons from Deploying an LLM-Based Bacalaureat Assessment Platform
- 用Gemini Flash模型自动批改近5年高考题
- 收集超100份学生作答，暴露评分不一致等问题
- 适合关注教育AI评估的开发者与研究者
获取高质量的罗马尼亚高考备考资源和反馈对偏远地区学生而言仍具挑战。本文介绍BacPrep平台，一个探索大语言模型（LLM）在自动化评估中潜力的实验性在线系统，旨在提供免费、可访问的备考支持。该平台使用过去5年的官方考题，采用最新可用的Gemini Flash模型（当前为Gemini 2.5 Flash，通过gemini-flash-latest接口调用），在数据收集阶段优先保障用户体验质量，并对模型版本进行锁定以供后续严格评估。平台已收集超过100份计算机科学与罗马尼亚语科目的学生作答，初步评估了大模型评分质量。结果揭示多项重大问题：多次运行间评分不一致、分数累加时出现算术错误、大提示上下文下性能下降、未能正确应用学科特定评分权重，以及评分与定性反馈内部矛盾。这些发现推动了新架构设计，包括按科目分解提示、设置专属评分模块及多轮运行取中位数策略。专家对照人工评分的验证仍是下一步关键任务。
原文摘要 · Abstract (English)
Accessing quality preparation and feedback for the Romanian Bacalaureat exam is challenging, particularly for students in remote or underserved areas. This paper presents BacPrep, an experimental online platform exploring Large Language Model (LLM) potential for automated assessment, aiming to offer a free, accessible resource. Using official exam questions from the last 5 years, BacPrep employs the latest available Gemini Flash model (currently Gemini 2.5 Flash, via the \texttt{gemini-flash-latest} endpoint) to prioritize user experience quality during the data collection phase, with model versioning to be locked for subsequent rigorous evaluation. The platform has collected over 100 student solutions across Computer Science and Romanian Language exams, enabling preliminary assessment of LLM grading quality. This revealed several significant challenges: grading inconsistency across multiple runs, arithmetic errors when aggregating fractional scores, performance degradation under large prompt contexts, failure to apply subject-specific rubric weightings, and internal inconsistencies between generated scores and qualitative feedback. These findings motivate a redesigned architecture featuring subject-level prompt decomposition, specialized per-subject graders, and a median-selection strategy across multiple runs. Expert validation against human-graded solutions remains the critical next step.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。