通过识别错误模式优化评分标准,提升大模型自动评分的准确性。
Confusion-Aware Rubric Optimization for LLM-based Automated Grading
- 用混淆矩阵拆分错误信号,分别修复不同类型的判错。
- 在教师教育和理工科数据集上,准确率显著优于现有方法。
- 适合需要高精度自动评分的教育评估场景。
大语言模型(LLM)用于自动评分时,精准且无歧义的评分标准至关重要。然而,人工编写提示词常因模型误解或领域特异性不足而效果不佳。现有自动化优化框架通常将独立、无结构的错误样本合并为单一更新步骤,导致“规则稀释”——冲突约束削弱模型判断逻辑。为此,我们提出混淆感知评分标准优化(CARO)框架,通过结构化分离错误信号,提升准确性和计算效率。CARO利用混淆矩阵将整体错误信号分解为不同模式,实现对特定误判模式的诊断与修复。通过生成针对性“修复补丁”并采用多样性感知选择机制,避免指导冲突,无需资源密集的嵌套优化循环。在教师教育与STEM数据集上的实证评估显示,CARO显著优于当前最优方法。结果表明,以手术式、模式特定修复替代混合错误聚合,可大幅提升自动评估的可扩展性与精度。
原文摘要 · Abstract (English)
Accurate and unambiguous guidelines are critical for large language model (LLM) based graders, yet manually crafting these prompts is often sub-optimal as LLMs can misinterpret expert guidelines or lack necessary domain specificity. Consequently, the field has moved toward automated prompt optimization to refine grading guidelines without the burden of manual trial and error. However, existing frameworks typically aggregate independent and unstructured error samples into a single update step, resulting in "rule dilution" where conflicting constraints weaken the model's grading logic. To address these limitations, we introduce Confusion-Aware Rubric Optimization (CARO), a novel framework that enhances accuracy and computational efficiency by structurally separating error signals. CARO leverages the confusion matrix to decompose monolithic error signals into distinct modes, allowing for the diagnosis and repair of specific misclassification patterns individually. By synthesizing targeted "fixing patches" for dominant error modes and employing a diversity-aware selection mechanism, the framework prevents guidance conflict and eliminates the need for resource-heavy nested refinement loops. Empirical evaluations on teacher education and STEM datasets demonstrate that CARO significantly outperforms existing SOTA methods. These results suggest that replacing mixed-error aggregation with surgical, mode-specific repair yields robust improvements in automated assessment scalability and precision.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。