提出BROS方法，在节省内存的同时保持高效优化性能。
BROS: Bias-Corrected Randomized Subspaces for Memory-Efficient Single-Loop Bilevel Optimization
- 在随机子空间中进行更新，并用径向基校正保证无偏估计
- 在标准假设下达到与精确方法相当的收敛速度，样本复杂度为O(ε⁻²)
- 适合大规模神经网络的超参数学习与数据重加权任务
随机双层优化（SBO）已成为深度学习中超参数学习、数据重加权、表征学习和数据混合优化的标准框架。现有精确单循环SBO方法和内存高效的代理SBO方法要么对大型低层神经网络造成严重内存压力，要么在标准假设下缺乏竞争力的收敛保证。本文提出BROS，一种内存高效的单循环SBO方法，其收敛速率阶与精确单循环SBO方法相同。BROS在随机子空间中执行低层和辅助更新，并采用Rademacher双探针校正以恢复无偏的海森-作用估计器。我们证明，在仅标准假设下，BROS保持了MA-SOBA的$\mathcal O(\varepsilon^{-2})$样本复杂度，用于寻找$\varepsilon$-平稳点。在超数据清洗、数据混合学习、超表征学习和ViT样本重加权实验中，BROS将峰值内存减少高达44.9%，同时性能接近全空间基线。
原文摘要 · Abstract (English)
Stochastic bilevel optimization (SBO) has become a standard framework for hyperparameter learning, data reweighting, representation learning, and data-mixture optimization in deep learning. Existing exact single-loop SBO methods and memory-efficient surrogate SBO methods either create severe memory pressure for large lower-level neural networks or lack competitive convergence guarantees under standard assumptions. In this paper, we propose BROS, a memory-efficient single-loop SBO method with the same convergence rate order as exact single-loop SBO methods. BROS performs lower and auxiliary updates in randomized subspaces with a Rademacher bi-probe correction that recovers an unbiased Hessian-action estimator. We prove that BROS preserves the $\mathcal O(\varepsilon^{-2})$ sample complexity of MA-SOBA for finding an $\varepsilon$-stationary point under only standard assumptions. Experiments on hyper-data cleaning, data-mixture learning, hyper-representation learning, and ViT sample reweighting show that BROS reduces peak memory by up to 44.9% while closely matching full-space baseline performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。