不同归一化方式会显著影响正则化回归的系数收缩，需谨慎选择。
The Choice of Normalization Influences Shrinkage in Regularized Regression
- 研究了归一化方法对lasso、岭回归和弹性网的影响机制。
- 二值特征比例失衡会改变系数估计，且与归一化/正则化组合相关。
- 建议用方差或标准差缩放二值特征，或调整惩罚权重以缓解偏差。
正则化模型对特征尺度敏感，因此通常在建模前对特征进行标准化（中心化并缩放）。然而，存在多种归一化方式，其选择可能对模型结果产生重大影响。尽管如此，该问题至今缺乏研究。本文首次系统探讨了在lasso、岭回归和弹性网回归中归一化的影响。重点关注二值特征，发现其类别平衡（1的比例）直接影响回归系数，且该效应取决于归一化与正则化方法的组合。我们证明，在lasso中使用方差缩放二值特征，或在岭回归中使用标准差缩放，可减轻此效应，但会增加系数估计的方差。对于弹性网，通过缩放惩罚权重可达到类似效果。此外，我们也初步研究了二值与连续特征混合、交互项等情况下的归一化策略。
原文摘要 · Abstract (English)
Regularized models are often sensitive to the scales of the features in the data and it has therefore become standard practice to normalize (center and scale) the features before fitting the model. But there are many different ways to normalize the features and the choice may have dramatic effects on the resulting model. In spite of this, there has so far been no research on this topic. In this paper, we begin to bridge this knowledge gap by studying normalization in the context of lasso, ridge, and elastic net regression. We focus on binary features and show that their class balances (proportions of ones) directly influences the regression coefficients and that this effect depends on the combination of normalization and regularization methods used. We demonstrate that this effect can be mitigated by scaling binary features with their variance in the case of the lasso and standard deviation in the case of ridge regression, but that this comes at the cost of increased variance of the coefficient estimates. For the elastic net, we show that scaling the penalty weights, rather than the features, can achieve the same effect. Finally, we also tackle mixes of binary and normal features as well as interactions and provide some initial results on how to normalize features in these cases.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。