提出可逆深度均衡模型，实现精确梯度计算与更稳定训练。
Reversible Deep Equilibrium Models
- 通过可逆结构设计，实现梯度的精确反向传播。
- 在语言建模和图像分类任务中优于对比的隐式与显式模型。
- 无需正则化，函数求值次数显著少于传统DEQs，适合高效训练。
深度均衡模型（DEQs）是一类隐式模型，其输出由一个学习函数的固定点隐式定义。这类模型通过将大量深层网络替换为迭代多次的单层结构，在大规模任务中表现出优于显式（固定深度）模型的性能。然而，传统DEQ的梯度计算是近似的，常导致训练不稳定，需依赖正则化或大量函数求值来修复。本文提出可逆深度均衡模型（RevDEQs），实现精确梯度计算，无需正则化，且函数求值次数远低于传统DEQs。实验表明，RevDEQ在语言建模和图像分类任务上显著优于同类隐式与显式模型。
原文摘要 · Abstract (English)
Deep Equilibrium Models (DEQs) are an interesting class of implicit model where the model output is implicitly defined as the fixed point of a learned function. These models have been shown to outperform explicit (fixed-depth) models in large-scale tasks by trading many deep layers for a single layer that is iterated many times. However, gradient calculation through DEQs is approximate. This often leads to unstable training dynamics and requires regularisation or many function evaluations to fix. Here, we introduce Reversible Deep Equilibrium Models (RevDEQs) that allow for exact gradient calculation, no regularisation and far fewer function evaluations than DEQs. We show that RevDEQs significantly improve performance on language modelling and image classification tasks against comparable implicit and explicit models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。