提出逻辑合规度评分,衡量模型是否遵守预设规则。
Beyond Accuracy: Measuring Logical Compliance of Predictive Models
- 设计RVS评分,区分硬规则与软规则,独立评估逻辑一致性。
- 相同准确率下,不同模型逻辑合规度差异显著。
- 适用于知识图谱、关系回归等场景,适合高风险领域研究者。
机器学习模型通常以排序质量、预测误差或分类准确率等指标评估,这些指标仅反映预测与真实值的接近程度,无法衡量输出是否符合预设逻辑或领域约束。在医疗、金融、自动驾驶等高风险应用中,逻辑一致性与准确性同等重要,但目前缺乏标准评估方法。本文提出规则违背评分(Rule Violation Score, RVS),用于量化预测模型对一组逻辑规则的遵守程度,且与预测准确率无关。RVS能区分硬规则(严格约束)与软规则(统计规律),适用于任意数据集和基于关系词汇表达的预测模型,并可通过自动生成的SQL查询计算霍恩规则。RVS不仅能评估模型,还可检测训练数据的逻辑一致性,识别定义不当的规则。我们在三个涵盖知识图谱链接预测与关系回归的基准上进行了测试,包含基于规则、嵌入式和神经符号的模型。结果表明,两个预测准确率相近的模型可能表现出显著不同的逻辑合规性,揭示了传统指标无法捕捉的模型行为差异。
原文摘要 · Abstract (English)
Machine learning models are predominantly evaluated through predictive performance metrics such as ranking quality, prediction error, or classification accuracy. While these metrics effectively quantify how closely predictions match the ground truth, they do not assess whether model outputs respect predefined logical or domain-specific constraints. In high-stakes applications, including healthcare, finance, and autonomous systems, logical consistency can be as critical as predictive accuracy, yet no standard metric captures this dimension. We introduce the Rule Violation Score (RVS), a complementary evaluation metric that quantifies the extent to which a predictive model respects a given set of logical rules, independently of predictive accuracy. RVS treats hard rules (strict constraints) and soft rules (statistical regularities) differently, can be evaluated on any dataset and on any predictive model expressed over a relational vocabulary, and can be computed using SQL queries that are automatically generated for Horn rules. Beyond evaluating models, RVS can also evaluate the logical consistency of training datasets and help identify poorly defined rules. We evaluate RVS on three benchmarks covering knowledge graph link prediction and relational regression, including rule-based, embedding-based, and neuro-symbolic predictive models. Our results demonstrate that two models achieving comparable predictive accuracy can exhibit substantially different levels of logical compliance, revealing differences in model behavior that standard metrics fail to capture.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。