用热启动和早停法高效估算变量重要性,适合大规模模型解释。
Reliable and scalable variable importance estimation via warm-start and early stopping
- 结合热启动与早停,避免重复训练提升效率
- 在真实数据与模拟中验证了计算速度与准确率双提升
- 理论保证适用于大宽神经网络与对称树梯度提升
随着黑箱预测模型日益普及,对其解释性的需求愈发重要。变量重要性与Shapley值是适用于任何预测模型的可解释性度量,用于评估变量对预测性能的贡献。当变量数量庞大时,重新训练神经网络等黑箱算法会带来巨大计算开销。本文针对基于梯度下降与梯度提升的方法(如神经网络、梯度提升决策树),提出一种可扩展的变量重要性估计方法。通过利用梯度方法的早停思想,并结合使用随机丢弃法进行热启动,实现了对任意可表示为迭代核更新方程的算法的高效估计。理论上,我们基于核方法的早停理论,为具有足够宽(但非无穷)宽度的神经网络及以对称树为弱学习器的梯度提升决策树提供了保证。仿真与真实数据实验表明,相比完全重训练,早停显著降低计算成本,且本方法准确性更高。
原文摘要 · Abstract (English)
As opaque black-box predictive models become more prevalent, the need to develop interpretations for these models is of great interest. The concept of variable importance and Shapley values are interpretability measures that applies to any predictive model and assesses how much a variable or set of variables improves prediction performance. When the number of variables is large, estimating variable importance presents a significant computational challenge because re-training neural networks or other black-box algorithms requires significant additional computation. In this paper, we address this challenge for algorithms using gradient descent and gradient boosting (e.g. neural networks, gradient-boosted decision trees). By using the ideas of early stopping of gradient-based methods in combination with warm-start using the dropout method, we develop a scalable method to estimate variable importance for any algorithm that can be expressed as an iterative kernel update equation. Importantly, we provide theoretical guarantees by using the theory for early stopping of kernel-based methods for neural networks with sufficiently large (but not necessarily infinite) width and gradient-boosting decision trees that use symmetric trees as a weaker learner. We also demonstrate the efficacy of our methods through simulations and a real data example which illustrates the computational benefit of early stopping rather than fully re-training the model as well as the increased accuracy of our approach.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。