提出新方法检测弱标签数据集中的捷径依赖，避免误判模型真能力。
Metadata Predictability Is Not Evidence Dependence: An Intervention-Based Audit for Weak-Label Benchmarks
- 结合元数据统计与证据干预测试，识别模型是否依赖元数据捷径。
- 在合成数据集上验证：元数据预测性中等但证据干预无反应，说明存在捷径。
- 建议评估时同步报告元数据筛查、证据干预和阅读器校准结果。
我们研究弱标签基准的协议级测试：当提供的证据被干预时，基准输出是否发生变化。仅依赖元数据的快捷方式检测回答的是不同问题，即输出是否可从元数据先验中预测。因此，我们将元数据统计量——元数据先验主导得分（MPDS）与证据干预统计量ΔEvi结合，后者衡量在跨项随机化下对证据身份的敏感性。合成HotpotQA提供了一个反例：MPDS仅为0.643，但ΔEvi为0。强阅读器重跑显示校准应纳入测试流程：SNLI出现校准反转，重构后的HotpotQA处于以问题为主导的警示区域，FEVER则在四个Transformer模型上均表现出强证据敏感性。实践启示明确：基准审计应同时报告元数据筛查、证据干预和阅读器强度校准结果。
原文摘要 · Abstract (English)
We study a protocol-level test for weak-label benchmarks: whether benchmark outputs change when the provided evidence is intervened on. Metadata-only shortcut checks answer a different question, namely whether outputs are predictable from metadata priors. We therefore combine a metadata statistic, the Metadata Prior Dominance Score (MPDS), with an evidence-intervention statistic, ΔEvi, measuring sensitivity to evidence identity under cross-item shuffling. Synthetic HotpotQA gives a constructed counterexample to metadata-only screening: MPDS is only moderate (0.643), yet ΔEvi is zero. Stronger-reader reruns show why calibration belongs in the test procedure: SNLI shows a calibration reversal, reconstructed HotpotQA occupies a question-dominant warning region, and FEVER is a strongly evidence-sensitive positive control across four transformers. The practical lesson is simple: benchmark audits should report metadata-only screening, evidence intervention, and reader-strength calibration together.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。