通过随机路由正则化，显著提升剪枝后模型的分割精度。
Reg4Pru: Regularisation Through Random Token Routing for Token Pruning
- 引入随机令牌路由作为训练正则化，缓解剪枝导致的性能下降。
- 在血管分割任务中，平均精度提升46%，速度加快29%。
- 适合需要高效推理且对精度要求高的视觉分割场景。
Transformer 在现代视觉模型中广泛应用，因其能随数据集规模扩展并具备良好泛化能力。然而其计算量随令牌数量呈二次增长，成为主要瓶颈。已有方法如令牌剪枝可提升效率，但会因保留表示不稳定导致深层预测性能下降。本文提出 Reg4Pru，一种基于随机令牌路由的训练正则化技术，用于缓解剪枝带来的性能损失。在 FIVES 血管分割数据集上的实验表明，使用 Reg4Pru 的模型相比无路由训练的同构模型，平均精度提升 46%；同时在实现 29% 相对运行时间加速的前提下，仍保持优异性能。结果表明，Reg4Pru 是一种有效的令牌压缩正则化方法。
原文摘要 · Abstract (English)
Transformers are widely adopted in modern vision models due to their strong ability to scale with dataset size and generalisability. However, this comes with a major drawback: computation scales quadratically to the total number of tokens. Numerous methods have been proposed to mitigate this. For example, we consider token pruning with reactivating tokens from preserved representations, but the increased computational efficiency of this method results in decreased stability from the preserved representations, leading to poorer dense prediction performance at deeper layers. In this work, we introduce Reg4Pru, a training regularisation technique that mitigates token-pruning performance loss for segmentation. We compare our models on the FIVES blood vessel segmentation dataset and find that Reg4Pru improves average precision by an absolute 46% compared to the same model trained without routing. This increase is observed using a configuration that achieves a 29% relative speedup in wall-clock time compared to the non-pruned baseline. These findings indicate that Reg4Pru is a valuable regulariser for token reduction strategies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。