提出BPQP框架,让优化层反向传播更快更高效。
BPQP: A Differentiable Convex Optimization Framework for Efficient End-to-End Learning
- 将反向传播重构为可解的二次规划问题,避开复杂雅可比计算
- 在真实和模拟数据上提速近10倍,整体执行时间大幅缩短
- 适合需要快速迭代的端到端学习场景,如大规模优化决策
数据驱动的决策过程越来越多地采用端到端可学习的深度神经网络。某些层的输出由数学优化问题的解决定,催生了可微优化层,支持梯度反向传播。然而,现实场景常涉及大规模数据和大量约束,带来显著挑战。现有方法多依赖隐式微分,需昂贵的雅可比矩阵计算,效率低下。本文提出BPQP,一种面向高效端到端学习的可微凸优化框架。通过利用KKT矩阵的结构特性,将反向传播重构成简化且解耦的二次规划问题,使第一类优化算法可用于梯度计算,兼容任意先进求解器。随着求解技术演进,BPQP能持续提升效率。在模拟与真实数据集上的实验表明,其整体执行时间通常比其他可微优化层快一个数量级,显著优于基线方法。
原文摘要 · Abstract (English)
Data-driven decision-making processes increasingly utilize end-to-end learnable deep neural networks to render final decisions. Sometimes, the output of the forward functions in certain layers is determined by the solutions to mathematical optimization problems, leading to the emergence of differentiable optimization layers that permit gradient back-propagation. However, real-world scenarios often involve large-scale datasets and numerous constraints, presenting significant challenges. Current methods for differentiating optimization problems typically rely on implicit differentiation, which necessitates costly computations on the Jacobian matrices, resulting in low efficiency. In this paper, we introduce BPQP, a differentiable convex optimization framework designed for efficient end-to-end learning. To enhance efficiency, we reformulate the backward pass as a simplified and decoupled quadratic programming problem by leveraging the structural properties of the KKT matrix. This reformulation enables the use of first-order optimization algorithms in calculating the backward pass gradients, allowing our framework to potentially utilize any state-of-the-art solver. As solver technologies evolve, BPQP can continuously adapt and improve its efficiency. Extensive experiments on both simulated and real-world datasets demonstrate that BPQP achieves a significant improvement in efficiency--typically an order of magnitude faster in overall execution time compared to other differentiable optimization layers. Our results not only highlight the efficiency gains of BPQP but also underscore its superiority over differentiable optimization layer baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。