用规则引擎约束大模型，让自动批改图示更可信。
Sketch2Feedback: Grammar-in-the-Loop Framework for Rubric-Aligned Feedback on Student STEM Diagrams
- 将批改流程拆解为感知、构图、校验、生成四步，用规则验证后才输出反馈。
- 在受控测试中，规则框架的纠错反馈更准确且幻觉率低。
- 适合教育技术开发者和希望提升批改可信度的研究者。
在STEM教育中，及时提供符合评分标准的图示反馈始终是难题。尽管大模态模型（LMMs）能联合解析图像并生成解释，但其易产生幻觉，削弱了教学应用中的可信度。我们提出Sketch2Feedback，一种语法在环框架，将问题分解为四个阶段：混合感知、符号图构建、约束检查与受限视觉语言模型反馈，确保语言模型仅对上游规则引擎验证过的错误进行表述。我们在两个合成微型基准上评估：FBD-10（受力图）和Circuit-10（电路图），各含500张图像，涵盖标准与高噪声增强层级。在每项基准100个测试样本上，使用95%自助置信区间分析，结果表明：Qwen2-VL-7B在受力图（0.570）和电路图（0.528）上取得最高微平均F1，但幻觉率极高（0.78, 0.98）。一个集成预言机通过每样本选取最优预测，达到F1=0.556，幻觉率0.320，显示出语法与端到端方法间的可利用互补性。设定置信阈值tau=0.7可使电路幻觉从0.970降至0.880，且不损失F1。高噪声增强揭示领域差异：受力图检测具韧性，而电路检测性能显著下降。基于大模型作为评判者的评估显示，语法管道产生的电路反馈更具可操作性（4.85/5），优于端到端模型（3.11/5）。所有代码、数据集与评估脚本均已开源。
原文摘要 · Abstract (English)
Providing timely, rubric-aligned feedback on student-drawn diagrams is a persistent challenge in STEM education. While large multimodal models (LMMs) can jointly parse images and generate explanations, their tendency to hallucinate undermines trust in classroom deployments. We present Sketch2Feedback, a grammar-in-the-loop framework that decomposes the problem into four stages -- hybrid perception, symbolic graph construction, constraint checking, and constrained VLM feedback -- so that the language model verbalizes only violations verified by an upstream rule engine. We evaluate on two synthetic micro-benchmarks, FBD-10 (free-body diagrams) and Circuit-10 (circuit schematics), each with 500 images spanning standard and hard noise augmentation tiers, comparing our pipeline against end-to-end LMMs (LLaVA-1.5-7B, Qwen2-VL-7B), a vision-only detector, a YOLOv8-nano learned detector, and an ensemble oracle. On n=100 test samples per benchmark with 95% bootstrap CIs, results are mixed and instructive: Qwen2-VL-7B achieves the highest micro-F1 on both FBDs (0.570) and circuits (0.528), but with extreme hallucination rates (0.78, 0.98). An ensemble oracle that selects the best prediction per sample reaches F1=0.556 with hallucination 0.320 on FBDs, demonstrating exploitable complementarity between grammar and end-to-end approaches. Confidence thresholding at tau=0.7 reduces circuit hallucination from 0.970 to 0.880 with no F1 loss. Hard noise augmentation reveals domain-dependent robustness: FBD detection is resilient while circuit detection degrades sharply. An LLM-as-judge evaluation confirms that the grammar pipeline produces more actionable circuit feedback (4.85/5) than the end-to-end LMM (3.11/5). We release all code, datasets, and evaluation scripts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。