提出可微分的低秩正则化近似方法，提升优化效率与适用性。
Efficient Differentiable Approximation of Generalized Low-rank Regularization
- 设计可微分的低秩正则化近似，避免奇异值分解计算瓶颈。
- 在多种任务中验证有效性，支持核范数、Schatten-p范数等广泛形式。
- 代码开源，适配深度学习框架，适合需要高效低秩约束的研究者。
低秩正则化（LRR）广泛应用于各类机器学习任务，但其优化极具挑战性——直接优化秩函数在一般情况下属于NP难问题。为克服此困难，已有研究提出多种秩函数松弛方法。然而，这些松弛形式的优化通常依赖奇异值分解（SVD），该操作耗时且不可微，无法使用基于梯度的方法进行优化。为此，本文提出一种高效可微分的广义低秩正则化近似方法。所考虑的正则化形式涵盖核范数、Schatten-p范数以及多种非凸松弛。该方法使LRR项可如插件般灵活加入损失函数，且基于GPU友好的操作实现高效便捷。同时，本文提供了收敛性分析，严格证明了所提秩估计器的偏差与方差随样本量和迭代步数增加而快速减小。实验表明，该方法在多个任务中表现出卓越的通用性与效率。代码已公开于 https://github.com/naiqili/EDLRR。
原文摘要 · Abstract (English)
Low-rank regularization (LRR) has been widely applied in various machine learning tasks, but the associated optimization is challenging. Directly optimizing the rank function under constraints is NP-hard in general. To overcome this difficulty, various relaxations of the rank function were studied. However, optimization of these relaxed LRRs typically depends on singular value decomposition, which is a time-consuming and nondifferentiable operator that cannot be optimized with gradient-based techniques. To address these challenges, in this paper we propose an efficient differentiable approximation of the generalized LRR. The considered LRR form subsumes many popular choices like the nuclear norm, the Schatten-$p$ norm, and various nonconvex relaxations. Our method enables LRR terms to be appended to loss functions in a plug-and-play fashion, and the GPU-friendly operations enable efficient and convenient implementation. Furthermore, convergence analysis is presented, which rigorously shows that both the bias and the variance of our rank estimator rapidly reduce with increased sample size and iteration steps. In the experimental study, the proposed method is applied to various tasks, which demonstrates its versatility and efficiency. Code is available at https://github.com/naiqili/EDLRR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。