揭示算法展开的梯度偏差根源，提出截断早期迭代可缓解问题。
Understanding the Curse of Unrolling
- 通过非渐近分析揭示展开计算中梯度初始发散的原因
- 早期截断能有效缓解偏差并降低内存消耗
- 双层优化中的热启动隐含截断机制，具实用价值
算法展开在机器学习中广泛应用，尤其在超参数优化和元学习中，通过微分迭代算法计算解映射的雅可比矩阵。尽管在理想条件下展开可得到渐近正确的雅可比，但近期研究发现，导数迭代初期可能偏离真实值，即存在‘展开诅咒’现象。本文提供非渐近分析，阐明该行为的成因，并识别影响其的关键算法因素。研究显示，截断导数计算的早期迭代可缓解此问题，同时减少内存开销。此外，我们证明双层优化中的热启动会自然诱导一种隐式截断，构成实际可行的解决方案。理论结果在代表性实例上通过数值实验得到验证。
原文摘要 · Abstract (English)
Algorithm unrolling is ubiquitous in machine learning, particularly in hyperparameter optimization and meta-learning, where Jacobians of solution mappings are computed by differentiating through iterative algorithms. Although unrolling is known to yield asymptotically correct Jacobians under suitable conditions, recent work has shown that the derivative iterates may initially diverge from the true Jacobian, a phenomenon known as the curse of unrolling. In this work, we provide a non-asymptotic analysis that explains the origin of this behavior and identifies the algorithmic factors that govern it. We show that truncating early iterations of the derivative computation mitigates the curse while simultaneously reducing memory requirements. Finally, we demonstrate that warm-starting in bilevel optimization naturally induces an implicit form of truncation, providing a practical remedy. Our theoretical findings are supported by numerical experiments on representative examples.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。