提出新型DEQ模型，理论保证收敛且提速近5倍。
Lipschitz Multiscale Deep Equilibrium Models: A Theoretically Guaranteed and Accelerated Approach
- 通过调整超参数重构架构，确保前向与反向传播的固定点收敛。
- 在CIFAR-10上实现最高4.75倍加速，准确率略有下降。
- 适合追求高效推理且可容忍微小精度损失的场景。
深度平衡模型（DEQ）通过神经网络层变换的不动点实现无限深度表示，无需堆叠层数，在多项大规模实验中表现接近当前最优方法，且内存占用显著更低。然而，由于每次输入都需反复执行无收敛保证的不动点迭代，训练和推理耗时远超传统方法。为此，本文提出一种重构模型架构的方法，通过超参数调节实现前向与反向传播的理论保证不动点收敛。所提出的利普希茨多尺度DEQ在图像分类任务中，于CIFAR-10上的数值实验实现了最高4.75倍加速，仅伴随轻微准确率下降。
原文摘要 · Abstract (English)
Deep equilibrium models (DEQs) achieve infinitely deep network representations without stacking layers by exploring fixed points of layer transformations in neural networks. Such models constitute an innovative approach that achieves performance comparable to state-of-the-art methods in many large-scale numerical experiments, despite requiring significantly less memory. However, DEQs face the challenge of requiring vastly more computational time for training and inference than conventional methods, as they repeatedly perform fixed-point iterations with no convergence guarantee upon each input. Therefore, this study explored an approach to improve fixed-point convergence and consequently reduce computational time by restructuring the model architecture to guarantee fixed-point convergence. Our proposed approach for image classification, Lipschitz multiscale DEQ, has theoretically guaranteed fixed-point convergence for both forward and backward passes by hyperparameter adjustment, achieving up to a 4.75$\times$ speed-up in numerical experiments on CIFAR-10 at the cost of a minor drop in accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。