发现人类标注差异的边界依赖于数据筛选，非普遍规律。
Selection Shapes the Boundary: A Preregistered Replication of Monotonicity and Label Agreement in Unselected NLI Populations
- 在未筛选的数据上复现原研究，使用相同标注器和四分类一致性指标。
- 所有对比均显示非单调项一致性更高，与原结论相反且效应量极小。
- 提醒研究者：基于筛选数据的结论需明确说明筛选条件。
以往关于自然语言推理中人类标注差异（HLV）的研究多依赖按分歧程度筛选的重标注资源。先前研究（arXiv:2607.15870）发现，包含非向上单调性算子的假设在ChaosNLI中标签一致性较低（Cliff's delta = -0.284），但仅限于多数标签恰好获得五票中的三票的样本。我们预先注册了对这一边界在原始数据集（SNLI和MultiNLI开发集）中的复现，采用相同算子标注器和四分类有序一致性结果。注册预测失败：七个对比全部得到正的Cliff's delta（非单调项一致性略高），唯一显著结果方向相反，所有效应量远低于预设最小关注效应量（0.10）。稳健性检验支持测量可靠性：模拟标注错误反而缩小效应，人工重标审校在200个新样本上达成四分类一致性0.875。结论认为，早期负向边界可能仅是低一致性筛选下的结构，并非总体特征，基于筛选重标注资源构建的HLV结论应明确声明其筛选条件。
原文摘要 · Abstract (English)
Prior work on human label variation (HLV) in natural language inference (NLI) has often relied on re-annotation resources that select items by disagreement level. An earlier study (arXiv:2607.15870) found that hypotheses containing non-upward monotonicity operators showed lower label agreement in ChaosNLI (Cliff's delta = -0.284), which is restricted to items whose majority label carries exactly three of five votes. We preregistered a replication of this boundary in the unselected populations that ChaosNLI was drawn from: the SNLI and MultiNLI development sets, using the same operator tagger and a four-level ordinal agreement outcome. The registered prediction fails. All seven contrasts return a positive Cliff's delta (non-upward items agree slightly more, not less), the only significant confirmatory contrast has the opposite sign to the registration, and every effect is far below our smallest effect size of interest (0.10). Robustness checks support the measurement: simulated tagger misclassification shrinks the effects rather than manufacturing them, and a manual re-tagging audit reaches four-class agreement of 0.875 on a fresh 200-item sample. We conclude that the earlier negative boundary is plausibly a structure conditional on low-agreement selection rather than a population-level property, and that HLV structure claims built on selected re-annotation resources should state their selection conditional explicitly.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。