提出一种新型双层优化算法,高效处理大规模图像修复中的超参数调优。
Bilevel Learning via Inexact Stochastic Gradient Descent
- 采用不精确随机梯度,动态调整求解精度与步长以平衡效率与收敛性
- 理论证明在期望下达到O(k^{-1/4})的收敛速率,实验验证其稳定性与准确性
- 适合需要高维超参数调优的大规模成像任务,如去噪与补全
双层优化是机器学习中高维超参数调优的核心工具,广泛应用于成像领域,例如学习数据自适应正则项和优化变分正则化中的前向算子。这些问题通常涉及大量数据和众多参数,需依赖基于随机梯度的算法。然而,关于参数的精确梯度(即超梯度)难以获取,且其精度通常与计算成本呈线性关系。因此,算法必须在无需过度精度的前提下高效求解。现有方法在上下两层均引入随机性以提升泛化能力,但通常固定内层迭代次数,与渐近收敛假设冲突。本文推进了不精确随机双层优化的理论研究,证明在衰减精度与步长调度下可实现收敛并给出速率:期望收敛速率为O(k^{-1/4})。在使用凸岭正则和输入凸网络的图像去噪与补全实验中验证分析结果:递减步长提升稳定性,精度调度比步长策略更关键,自适应预条件(如Adam)进一步提升性能。这些成果连接了理论与实践,为大规模成像问题提供了收敛保证与实用指导。
原文摘要 · Abstract (English)
Bilevel optimization is a central tool in machine learning for high-dimensional hyperparameter tuning. Its applications are vast; for instance, in imaging it can be used for learning data-adaptive regularizers and optimizing forward operators in variational regularization. These problems are large in many ways: a lot of data is usually available to train a large number of parameters, calling for stochastic gradient-based algorithms. However, exact gradients with respect to parameters (so-called hypergradients) are not available, and their precision is usually linearly related to computational cost. Hence, algorithms must solve the problem efficiently without unnecessary precision. The design of such methods is still not fully understood, especially regarding how accuracy requirements and step size schedules affect theoretical guarantees and practical performance. Existing approaches introduce stochasticity at both the upper level (e.g., in sampling or mini-batch estimates) and the lower level (e.g., in solving the inner problem) to improve generalization, but they typically fix the number of lower-level iterations, which conflicts with asymptotic convergence assumptions. In this work, we advance the theory of inexact stochastic bilevel optimization. We prove convergence and establish rates under decaying accuracy and step size schedules, showing that with optimal configurations convergence occurs at an $\mathcal{O}(k^{-1/4})$ rate in expectation. Experiments on image denoising and inpainting with convex ridge regularizers and input-convex networks confirm our analysis: decreasing step sizes improve stability, accuracy scheduling is more critical than step size strategy, and adaptive preconditioning (e.g., Adam) further boosts performance. These results bridge theory and practice, providing convergence guarantees and practical guidance for large-scale imaging problems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。