让视觉Transformer训练更平滑，推理时仍高效。
Training Noise Token Pruning
- 用连续噪声替代离散丢弃，训练更稳定。
- ImageNet上优于现有剪枝方法，精度损失小。
- 适合追求高效部署的视觉模型研究者。
本文提出训练噪声标记剪枝（Training Noise Token, TNT）用于视觉Transformer。该方法将离散的标记丢弃条件放松为连续的加性噪声，使训练过程优化更平滑，同时在部署阶段保持离散丢弃带来的计算优势。我们建立了与率失真理论的理论联系，并在ViT和DeiT架构上基于ImageNet数据集进行了实证评估，结果表明TNT在性能上优于先前的剪枝方法。
原文摘要 · Abstract (English)
In the present work we present Training Noise Token (TNT) Pruning for vision transformers. Our method relaxes the discrete token dropping condition to continuous additive noise, providing smooth optimization in training, while retaining discrete dropping computational gains in deployment settings. We provide theoretical connections to Rate-Distortion literature, and empirical evaluations on the ImageNet dataset using ViT and DeiT architectures demonstrating TNT's advantages over previous pruning methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。