提出无需知道链接函数的稀疏二值GLM学习算法,高效且最优。
Learning sparse generalized linear models with binary outcomes via iterative hard thresholding
- 用迭代硬阈值法直接优化ReLU损失,不依赖链接函数
- 在逻辑回归中达到最优样本复杂度,与理论下界一致
- 适用于高维稀疏数据,适合机器学习与统计建模场景
广义线性模型(GLMs)广泛用于建模数据,尤其在二值结果(如逻辑回归、正态概率回归)中表现优异。现代数据常为高维但内在维度低,稀疏性约束合理可行。本文提出一种基于迭代硬阈值的算法(BIHT),用于稀疏二值GLMs的参数估计。该算法不需预先知道链接函数,具有通用性。理论上证明其统计高效,可收敛至正确解。在逻辑回归中,样本复杂度达到最优(与已知下界仅差对数因子),是首个在所有噪声环境下兼具计算效率与统计最优性的算法;在正态概率回归中,样本复杂度与逻辑回归同阶。
原文摘要 · Abstract (English)
In statistics, generalized linear models (GLMs) are widely used for modeling data and can expressively capture potential nonlinear dependence of the model's outcomes on its covariates. Within the broad family of GLMs, those with binary outcomes, which include logistic and probit regressions, are motivated by common tasks such as binary classification with (possibly) non-separable data. In addition, in modern machine learning and statistics, data is often high-dimensional yet has a low intrinsic dimension, making sparsity constraints in models another reasonable consideration. In this work, we propose to use and analyze an iterative hard thresholding (projected gradient descent on the ReLU loss) algorithm, called binary iterative hard thresholding (BIHT), for parameter estimation in sparse GLMs with binary outcomes. We establish that BIHT is statistically efficient and converges to the correct solution for parameter estimation in a general class of sparse binary GLMs. Unlike many other methods for learning GLMs, including maximum likelihood estimation, generalized approximate message passing, and GLM-tron (Kakade et al. 2011; Bahmani et al. 2016), BIHT does not require knowledge of the GLM's link function, offering flexibility and generality in allowing the algorithm to learn arbitrary binary GLMs. As two applications, logistic and probit regression are additionally studied. In this regard, it is shown that in logistic regression, the algorithm is in fact statistically optimal in the sense that the order-wise sample complexity matches (up to logarithmic factors) the lower bound obtained previously. To the best of our knowledge, this is the first work achieving statistical optimality for logistic regression in all noise regimes with a computationally efficient algorithm. Moreover, for probit regression, our sample complexity is on the same order as that obtained for logistic regression.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。