用记忆增强Transformer学会所有线性一阶优化方法
Toward generalizable learning of all (linear) first-order methods via memory augmented Transformers
- 用记忆增强Transformer模拟梯度下降及更复杂算法
- 在新数据上通过专家混合实现测试时自适应，提升泛化能力
- 将优化器本身作为可学习模型，数据驱动提升性能
我们证明，记忆增强的Transformer能够实现整个线性一阶方法（LFOMs）类，该类包含梯度下降（GD）、共轭梯度下降（CGD）、动量法及其各种线性组合变体。基于先前研究中Transformer模拟GD的工作，本文提供了理论与实证证据，表明记忆增强Transformer可学习更先进的优化算法。随后，我们首次尝试将学习到的算法转化为实际可用方法，提出一种测试时自适应的专家混合（MoE）策略，以应对分布外（OOD）样本。最后，我们证明LFOMs本身可作为可学习算法，其参数可通过数据训练获得优异性能。
原文摘要 · Abstract (English)
We show that memory-augmented Transformers can implement the entire class of linear first-order methods (LFOMs), a class that contains gradient descent (GD) and more advanced methods such as conjugate gradient descent (CGD), momentum methods and all other variants that linearly combine past gradients. Building on prior work that studies how Transformers simulate GD, we provide theoretical and empirical evidence that memory-augmented Transformers can learn more advanced algorithms. We then take a first step toward turning the learned algorithms into actually usable methods by developing a mixture-of-experts (MoE) approach for test-time adaptation to out-of-distribution (OOD) samples. Lastly, we show that LFOMs can themselves be treated as learnable algorithms, whose parameters can be learned from data to attain strong performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。