揭示视觉Transformer无训练剪枝崩溃根源，提出更稳定的替代方案
Why Training-Free Token Reduction Collapses: The Inherent Instability of Pairwise Scoring Signals
- 发现成对相似性信号在深层网络中严重失真，导致压缩性能骤降
- 新方法CATIS用单值信号提升稳定性，在63%算力削减下保持96.9%准确率
- 适合关注模型压缩鲁棒性的研究者和工程落地开发者
无训练的视觉Transformer token 剪枝方法（ToMe、ToFu、PiToMe、MCTF）虽采用不同评分机制，却在高压缩比下表现出近乎一致的陡峭性能坍塌。本文通过构建诊断框架，引入排名一致性ρ_s与非对角相关性ρ_off，将坍塌分解为两部分：(1) 层级剪枝固有的信号无关误差放大，预测出凸型帕累托曲线，且临界压缩率r_crit ∝ 1/L；(2) 共同依赖成对相似性信号，其排名一致性从ρ_s=0.88降至深层的0.27。成对排序受O(N_p²)联合扰动影响，远不如单值信号的O(N_p)稳定（中心极限定理保障）。基于此诊断，提出CATIS：使用单值信号提高触发阈值，引入筛选机制抑制增益。在ViT-Large上实现63% FLOPs削减时，保留81.0%原精度的96.9%（即76.8%），而所有基线方法坍缩至43–65%。
原文摘要 · Abstract (English)
Training-free token reduction methods for Vision Transformers (ToMe, ToFu, PiToMe, and MCTF) employ different scoring mechanisms, yet they share a closely matched cliff-like collapse at high compression. This paper explains \emph{why}. We develop a diagnostic framework with two tools, ranking consistency $ρ_s$ and off-diagonal correlation $ρ_\text{off}$, that decomposes the collapse into (1)a signal-agnostic error amplifier inherent to layer-wise reduction, predicting convex Pareto curves and $r_{\text{crit}} \propto 1/L$; and (2)shared reliance on \emph{pairwise} similarity signals whose ranking consistency degrades from $ρ_s{=}0.88$ to $0.27$ in deep layers. Pairwise rankings are inherently unstable ($O(N_p^2)$ joint perturbations) while unary signals enjoy greater stability ($O(N_p)$ perturbations, CLT). From three design principles derived from this diagnosis, we construct CATIS as a constructive validation: unary signals raise the trigger threshold, triage suppresses the gain. On ViT-Large at 63% FLOPs reduction, CATIS retains 96.9% of vanilla accuracy (81.0%) on ImageNet-1K where all baselines collapse to 43--65%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。