提出一种新正则化方法，可让优化解具备离散、正交等特性。
Cauchy-Schwarz Regularizers
- 基于柯西-施瓦茨不等式构造可微正则项
- 在欠定线性系统求解与神经网络权重量化中效果显著
- 自动适应尺度，适合梯度法和大规模优化
我们提出一类新型正则化函数——柯西-施瓦茨（Cauchy-Schwarz, CS）正则化，可用于诱导优化问题解向量的多种性质。为展示其通用性，我们推导出能促进离散取值向量、某矩阵特征向量及正交矩阵的正则化函数。所得CS正则化项形式简洁、可微，且可无虚假驻点，适用于梯度基求解器与大规模优化。此外，该正则化自动适应合适尺度，在神经网络权重量化中尤为有益。通过求解欠定线性方程组与权重量化任务验证了其有效性。我们还讨论了特殊形式、变体与推广，进一步拓展出更广泛的新正则化类。
原文摘要 · Abstract (English)
We introduce a novel class of regularization functions, called Cauchy-Schwarz (CS) regularizers, which can be designed to induce a wide range of properties in solution vectors of optimization problems. To demonstrate the versatility of CS regularizers, we derive regularization functions that promote discrete-valued vectors, eigenvectors of a given matrix, and orthogonal matrices. The resulting CS regularizers are simple, differentiable, and can be free of spurious stationary points, making them suitable for gradient-based solvers and large-scale optimization problems. In addition, CS regularizers automatically adapt to the appropriate scale, which is, for example, beneficial when discretizing the weights of neural networks. To demonstrate the efficacy of CS regularizers, we provide results for solving underdetermined systems of linear equations and weight quantization in neural networks. Furthermore, we discuss specializations, variations, and generalizations, which lead to an even broader class of new and possibly more powerful regularizers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。