让最小二乘法变成可微分模块，解锁新应用
Matrix-Free Least Squares Solvers: Values, Gradients, and What to Do With Them
- 推导自定义梯度，使求解器可嵌入神经网络
- 在5000万参数模型上实现权重稀疏化
- 适合需要可微线性代数的机器学习研究者
本文认为最小二乘法在现代机器学习中具有巨大未被发掘的潜力，远不止用于拟合线性模型。为释放其潜能，我们推导了定制梯度，将求解器转化为可微算子，如同神经网络层一般，从而支持多种应用。实证表明：(i) 在5000万参数模型上实现权重稀疏化；(ii) 在基于得分的生成模型中施加保守性约束；(iii) 根据预测性能对高斯过程进行超参数调优。本工作代表了可微线性代数工具发展的下一阶段，使其更广泛地服务于机器学习从业者。
原文摘要 · Abstract (English)
This paper argues that the method of least squares has significant unfulfilled potential in modern machine learning, far beyond merely being a tool for fitting linear models. To release its potential, we derive custom gradients that transform the solver into a differentiable operator, like a neural network layer, enabling many diverse applications. Empirically, we demonstrate: (i) scalability by enforcing weight sparsity on a 50 million parameter model; (ii) imposing conservativeness constraints in score-based generative models; and (iii) hyperparameter tuning of Gaussian processes based on predictive performance. By doing this, our work represents the next iteration in developing differentiable linear-algebra tools and making them widely accessible to machine learning practitioners.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。