让Transformer每层都像优化算法一样降低损失，提升鲁棒性和泛化能力。
A Constrained Optimization Perspective of Unrolled Transformers
- 用约束优化替代传统训练，强制每层损失递减
- 在视频去噪和文本分类中保持原性能，提升抗扰动能力
- 适合追求模型稳定性和泛化性的研究者
我们提出一种约束优化框架来训练Transformer，使其行为类似优化下降算法。具体地，对目标函数施加逐层下降约束，并将标准的经验风险最小化（ERM）替换为原始-对偶训练方案。该方法使模型的中间表示在期望上随层数单调减少损失。我们在未展开的Transformer架构及常规预训练Transformer上，应用于视频去噪和文本分类任务。在这些设置下，约束型Transformer展现出更强的抗扰动鲁棒性，维持更高分布外泛化能力，同时保持分布内性能。
原文摘要 · Abstract (English)
We introduce a constrained optimization framework for training transformers that behave like optimization descent algorithms. Specifically, we enforce layerwise descent constraints on the objective function and replace standard empirical risk minimization (ERM) with a primal-dual training scheme. This approach yields models whose intermediate representations decrease the loss monotonically in expectation across layers. We apply our method to both unrolled transformer architectures and conventional pretrained transformers on tasks of video denoising and text classification. Across these settings, we observe constrained transformers achieve stronger robustness to perturbations and maintain higher out-of-distribution generalization, while preserving in-distribution performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。