证明了在无分布假设下，无法可靠判断模型类是否全错。
Are all models wrong? Fundamental limits in distribution-free empirical model falsification
- 提出一种无需依赖数据分布的通用方法，检验模型类最优性能下限。
- 证明对任意模型类，都存在无法确定其最低误差下界的本质困难。
- 适用于树模型和线性回归，警示过度自信的模型评估风险。
在统计与机器学习中，训练模型时通常希望所选模型类至少包含一个准确模型，即需对模型类风险（类内可达到的最低风险）建立上界。然而，同样重要的是建立风险下界，以判断拟合模型是否接近类内最优，或模型类是否不适合特定任务。尤其在插值学习场景中，模型被训练至在训练数据上实现零误差，此时我们不禁要问：能否至少证明模型类风险存在正的下界？还是说‘所有模型都是错的’根本无法被检测？本文在无分布假设下，通过建立模型无关的基本困难性结果，揭示了构造模型类最优测试误差下界的本质局限，并分析其对树模型和线性回归等具体模型类的影响。
原文摘要 · Abstract (English)
In statistics and machine learning, when we train a fitted model on available data, we typically want to ensure that we are searching within a model class that contains at least one accurate model -- that is, we would like to ensure an upper bound on the model class risk (the lowest possible risk that can be attained by any model in the class). However, it is also of interest to establish lower bounds on the model class risk, for instance so that we can determine whether our fitted model is at least approximately optimal within the class, or, so that we can decide whether the model class is unsuitable for the particular task at hand. Particularly in the setting of interpolation learning where machine learning models are trained to reach zero error on the training data, we might ask if, at the very least, a positive lower bound on the model class risk is possible -- or are we unable to detect that "all models are wrong"? In this work, we answer these questions in a distribution-free setting by establishing a model-agnostic, fundamental hardness result for the problem of constructing a lower bound on the best test error achievable over a model class, and examine its implications on specific model classes such as tree-based methods and linear regression.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。