提出混合模式微分方法，显著提升元学习的计算效率。
Scalable Meta-Learning via Mixed-Mode Differentiation
- 采用混合模式微分构建更高效的计算图
- 内存占用降低10倍以上，运行时间减少25%
- 适合大规模元学习任务的高效实现
基于梯度的双层优化在超参数优化、任务适配、算法发现及元学习中具有广泛应用。其通常需对梯度优化过程本身求导，导致复杂的二阶与混合导数计算，开销巨大。尽管现代自动微分库可便捷编写此类导数程序，但往往无法充分利用问题的特定结构，造成性能不佳。本文分析了此类情况，提出混合流元梯度（MixFlow-MG）——一种利用混合模式微分构造更高效可扩展计算图的实际算法，在现代元学习设置中，相比标准实现，内存消耗降低超过10倍，运行时间最多减少25%。
原文摘要 · Abstract (English)
Gradient-based bilevel optimisation is a powerful technique with applications in hyperparameter optimisation, task adaptation, algorithm discovery, meta-learning more broadly, and beyond. It often requires differentiating through the gradient-based optimisation itself, leading to "gradient-of-a-gradient" calculations with computationally expensive second-order and mixed derivatives. While modern automatic differentiation libraries provide a convenient way to write programs for calculating these derivatives, they oftentimes cannot fully exploit the specific structure of these problems out-of-the-box, leading to suboptimal performance. In this paper, we analyse such cases and propose Mixed-Flow Meta-Gradients, or MixFlow-MG -- a practical algorithm that uses mixed-mode differentiation to construct more efficient and scalable computational graphs yielding over 10x memory and up to 25% wall-clock time improvements over standard implementations in modern meta-learning setups.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。