CTM可能检测不到重大数据分布变化,因p值仍均匀。
Conformal Blindness: A Note on $A$-Cryptic change-points
- 用理想预测器构造出伪装成均匀p值的突变点
- 即使边际均值大幅改变,p值仍保持均匀分布
- 提醒安全系统需区分预测与诊断目标
Conformal Test Martingales(CTMs)是检验数据交换性假设的标准方法,通过监测p值序列的均匀性偏差来实现。尽管交换性蕴含均匀p值,但反之不成立。本文证明了存在显著交换性破坏却使p值仍均匀的情况,即“共形盲区”。通过理想“预测预言者”一致性度量(基于真实条件密度),我们构造出一种$A$-隐匿突变点。在二元高斯分布中,沿特定直线改变边缘均值不会影响一致性得分分布,导致p值完全均匀。模拟显示,即使发生巨大分布偏移,也能对CTM完全隐蔽。这揭示了方法的根本局限,并强调一致性度量必须与潜在变化方向对齐。对比最优检测得分结果,凸显在安全关键系统中,有效性监控需严格区分预测与诊断目标。
原文摘要 · Abstract (English)
Conformal Test Martingales (CTMs) are a standard method within the Conformal Prediction framework for testing the crucial assumption of data exchangeability by monitoring deviations from uniformity in the p-value sequence. Although exchangeability implies uniform p-values, the converse does not hold. This raises the question of whether a significant break in exchangeability can occur, such that the p-values remain uniform, rendering CTMs blind. We answer this affirmatively, demonstrating the phenomenon of \emph{conformal blindness}. Through explicit construction, for the theoretically ideal ``predictive oracle'' conformity measure (given by the true conditional density), we demonstrate the possibility of an \emph{$A$-cryptic change-point} (where $A$ refers to the conformity measure). Using bivariate Gaussian distributions, we identify a line along which a change in the marginal means does not alter the distribution of the conformity scores, thereby producing perfectly uniform p-values. Simulations confirm that even a massive distribution shift can be perfectly cryptic to the CTM, highlighting a fundamental limitation and emphasising the critical role of the alignment of the conformity measure with potential shifts. By contrasting the predictive oracle with recent results on detection-optimal scores, we emphasise that validity monitoring in safety-critical systems requires careful separation of predictive and diagnostic goals.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。