用符号增强提升模型对科学量值等价转换的纠错能力
Symbolic Augmentation Closes a Canonical-Equivalence Blind Spot in Neural Fact-Checkers
- 设计五类量值错误分类体系,构建1500条标注数据集
- 在量值等价转换场景下,准确率从36.5%提升至98.2%
- 适合关注科学文本可信度、模型鲁棒性的研究者
大语言模型在总结科学文本时常产生数值与单位幻觉,可能导致科学主张被无声反转。本文将错误检测重构为类型化验证问题,提出五类量值错误分类体系,并构建包含1500项样本的基准数据集,来源为PMC和arXiv,由两名独立的LLM标注员标注并经仲裁(Krippendorff's alpha = 0.882)。基于该数据集微调的ModernBERT编码器达到macro-F1 = 0.899,远超现成神经事实核查器;但四项探测显示其在物理等价量值的规范等价重写(如95°C与368.15 K)上准确率骤降至36.5%。为此提出符号增强训练框架,在训练阶段反向运行符号验证模块生成标签保持的增强数据。该方法使规范等价鲁棒性提升至98.2%,且分布内准确率小幅提升(macro-F1: 0.899 → 0.902),性能媲美闭源前沿大模型而无推理开销,并成功迁移到SciFact-Open外部基准(binary macro-F1: 0.791 → 0.828)。两项负面结果进一步强化结论:符号特征作为辅助输入无效,符号银标在教师噪声下反而恶化。综合表明,训练阶段的增强是符号与学习组件融合的最优路径。
原文摘要 · Abstract (English)
Large language models hallucinate numbers and units when summarizing scientific text, a failure mode that can silently invert a scientific claim. We recast the detection of such errors as typed verification: we introduce a five-class typed-quantity error taxonomy and a 1500-item benchmark, rewritten from PMC and arXiv sources and labeled by two independent LLM annotators with adjudication (Krippendorff's alpha = 0.882). A ModernBERT encoder fine-tuned on this benchmark reaches macro-F1 = 0.899, far above any off-the-shelf neural fact-checker, yet four probes expose a sharp structural blind spot: on canonical-equivalent rewrites of physically equivalent quantities (e.g., 95°C and 368.15 K) its accuracy collapses to 36.5%. We propose Symbolic Augmentation, a training-time framework that runs the modules of a symbolic verifier in reverse to generate label-preserving augmented training data. The augmentation lifts canonical-equivalence robustness to 98.2% while slightly improving in-distribution accuracy (macro-F1: 0.899 to 0.902); the augmented encoder matches a closed-frontier LLM at no inference cost and transfers to an external benchmark (SciFact-Open binary macro-F1: 0.791 to 0.828). Two negative results sharpen the claim: symbolic features as auxiliary encoder inputs add nothing, and symbolic silver labels scale negatively under teacher noise. Together these results identify training-time augmentation as the right integration point between symbolic and learned components.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。