对比循环视觉变压器在参数与算力约束下的表现,揭示其优劣场景。
Training Crossroads for Recurrent Vision Transformers: Recurrence, Neural ODEs, and Deep Supervision

- 用统一协议比较循环、神经ODE与深度监督三种训练方式
- 参数受限时循环结构更优,算力受限时标准ViT仍占优
- 高阶求解器带来架构偏差而非精度提升,深度监督增强长期鲁棒性
视觉变压器(ViTs)虽性能强劲,但独立参数化每层会导致参数量随深度线性增长。单块循环视觉变压器(bViT)通过重复使用同一共享模块消除该增长。本文不提出新架构,而是固定bViT,在统一的CIFAR-100实验设置下,系统评估三种训练与推理范式:(i) 循环结构何时优于独立参数化深度——在匹配计算量(FLOPs)或匹配参数量(内存)条件下?(ii) 当残差循环模块通过微分方程求解器训练时,求解器阶数是数值优化工具还是引入架构偏见?(iii) 超出训练范围的鲁棒性代价多少?结果表明:当计算量为首要约束时,标准ViT更优;而参数量受限时,循环结构表现更好。残差网络被视作欧拉离散化的常理,在连续时间下对应状态减法向量场 $\ar{z}=F_θ(z)-z$;尽管原理已知,但若将模块封装为黑箱向量场,此区别易被忽略,我们量化了少数精度点上的代价。由于向量场与求解器联合学习,高阶求解器更像引入架构偏见而非提升数值精度,增益不均一。阶段式深度监督刻画出准确率-鲁棒性边界:不提升名义准确率,但在训练范围外仍能平稳退化,而朴素循环结构则迅速坍缩至近随机水平。
原文摘要 · Abstract (English)
Vision Transformers (ViTs) achieve strong image-recognition performance, but their parameter count grows linearly with depth when each block is independently parameterized. Single-block recurrent ViTs (bViT) remove this growth by repeatedly applying one shared block. Rather than proposing a new architecture, we fix a bViT and provide a controlled empirical characterization of three training and inference regimes under a common CIFAR-100 protocol, asking: (i)~when does recurrence beat independently parameterized depth---at matched FLOPs or at matched parameter memory? (ii)~when a residual recurrent block is trained through an ODE solver, does solver order act as numerical refinement or as an architectural bias? and (iii)~what does robustness beyond the training horizon cost in nominal accuracy? We find that standard ViTs remain preferable when FLOPs are the primary constraint, whereas recurrent ViTs offer a better accuracy--parameter trade-off under memory constraints. Consistent with the standard view of residual networks as Euler discretizations of ODEs, the continuous-time analogue of a residual recurrent block is the state-subtracted vector field $\dot{z}=F_θ(z)-z$; although known in principle, this distinction is easy to violate when the block is wrapped as a black-box vector field, and we qualify the cost at few accuracy points. Because the vector field is learned jointly with the solver, higher-order solvers act as a solver-induced architectural bias rather than a numerical-accuracy improvement, and their gains are not uniform. Finally, stage-wise deep supervision traces an accuracy--robustness frontier: it does not improve nominal accuracy, but degrades gracefully far beyond the training horizon, where naive recurrence collapses to near-random performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。