提出新方法精准估计模型在分布外数据的表现
ODD: Overlap-aware Estimation of Model Performance under Distribution Shift
- 基于重叠区域感知的分歧差异，避免源目标域重叠区的竞争
- 在多个基准上误差估计更准，且保持可靠性
- 适合需要可靠性能评估的部署场景
可靠准确地估计机器学习模型在未见测试域上的误差，是保障智能系统安全的关键问题。已有方法使用分歧差异（DIS²）在分布偏移下推导实用的误差界，通过在目标域上优化出最大分歧分类器来界定源分类器的误差。尽管该方法具有较高可靠性与竞争力，但存在一个问题：其分歧目标在源与目标域的重叠区域中产生竞争。我们提出一个直观假设：在重叠区域，目标域的分歧不应超过源域分歧（因支持度足够高），据此设计了重叠感知分歧差异（ODD）。ODD仅需在非重叠目标域中最大化分歧，从而消除竞争。我们的方法利用域分类器估计域重叠，相比DIS²能更优预测目标性能。我们在多种基准上进行了实验，结果表明，该方法在保持有效性的同时显著降低整体性能估计误差。代码与结果已开源。
原文摘要 · Abstract (English)
Reliable and accurate estimation of the error of an ML model in unseen test domains is an important problem for safe intelligent systems. Prior work uses disagreement discrepancy (DIS^2) to derive practical error bounds under distribution shifts. It optimizes for a maximally disagreeing classifier on the target domain to bound the error of a given source classifier. Although this approach offers a reliable and competitively accurate estimate of the target error, we identify a problem in this approach which causes the disagreement discrepancy objective to compete in the overlapping region between source and target domains. With an intuitive assumption that the target disagreement should be no more than the source disagreement in the overlapping region due to high enough support, we devise Overlap-aware Disagreement Discrepancy (ODD). Maximizing ODD only requires disagreement in the non-overlapping target domain, removing the competition. Our ODD-based bound uses domain-classifiers to estimate domain-overlap and better predicts target performance than DIS^2. We conduct experiments on a wide array of benchmarks to show that our method improves the overall performance-estimation error while remaining valid and reliable. Our code and results are available on GitHub.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。