研究离线存储与在线计算的权衡,给出实现精度所需的最小内存量
Memory-Computation Tradeoffs in Semi Amortized Parametric Optimization

- 用离线存储历史解作为启动点,结合固定步数梯度下降求解新问题
- 对强凸目标,给出精度ε下所需内存的上下界,对增长条件β>2的目标有近似界
- 揭示加速所需内存由优化器收敛率和解映射敏感性决定,适合系统设计者参考
学习型决策系统常通过离线数据或计算来降低在线计算成本。尽管此类方法在实践中取得成功,但对在固定在线计算预算下达到特定精度所需离线信息量缺乏普遍理解。本文从摊销参数优化视角研究该问题:离线阶段存储有限数量的历史问题解,线上阶段通过检索一个热启动并执行K步投影梯度下降求解新问题。针对紧致域上的光滑凸参数优化问题,采用基于存储解构建的非参数预测器进行分析。对于μ-强凸目标,建立了在固定在线迭代预算K下保证ε-精度所需的内存的匹配上下界;对于满足β-增长条件(β>2)的凸目标,获得近似匹配界,并识别出当K超过某一临界值后,增加内存不再带来收益的相变现象。进一步提出通用证明框架,(i) 显式量化加速的内存代价——即相对于无辅助的在线优化器实现指定加速所需多少离线内存;(ii) 识别出驱动该代价的两个关键量:在线优化器的收敛速率和解映射对问题参数的Lipschitz敏感性。在参数化岭回归上的实验验证了预测的内存-计算-精度权衡关系。
原文摘要 · Abstract (English)
Learning-enabled decision systems often use offline data or computation to reduce online compute cost. Despite the empirical success of such approaches, there is limited general understanding of how much offline information is needed to achieve a desired accuracy under a fixed online computation budget. We study this question through the lens of amortized parametric optimization: an offline phase stores a finite memory of solved problem instances, and an online phase produces a solution to a new instance by retrieving a warm start and applying $K$ steps of projected gradient descent. We analyze this setup for smooth convex parametric optimization over a compact domain, using a nonparametric predictor built from the stored offline solutions. For $μ$-strongly convex objectives, we establish matching upper and lower bounds on the memory required to guarantee $\varepsilon$-accuracy under a fixed online iteration budget $K$. For convex objectives satisfying a $β$-growth condition ($β>2$), we obtain near-matching bounds and identify a phase transition in $K$ beyond which additional memory provides no benefit. We further provide a general proof framework that (i) explicitly quantifies the memory cost of acceleration---how much offline memory is required to achieve a prescribed speedup over the unaided online optimizer---and (ii) identifies two key quantities driving this cost: the convergence rate of the online optimizer and the Lipschitz sensitivity of the solution map to the problem parameter. Experiments on parameterized ridge regression confirm the predicted memory--computation--accuracy tradeoffs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。