用稀疏自编码器实现视觉Transformer注意力头的可控动态剪枝
Steering Sparse Autoencoder Latents to Control Dynamic Head Pruning in Vision Transformers (Student Abstract)
- 通过稀疏自编码器解耦特征，生成可解释的稀疏潜在变量
- 针对不同类别定向调控潜变量，使剪枝后准确率提升至82%且头使用率降至0.33
- 适用于需要高效且可解释剪枝的视觉模型部署场景
视觉变压器（ViTs）中的动态注意力头剪枝能提升效率，但现有策略难以解释和控制。本文提出一种新框架，将稀疏自编码器（SAEs）与动态剪枝结合，利用其将密集嵌入分解为可解释、可操控的稀疏潜变量的能力。具体地，在ViT最后一层残差嵌入上训练SAE，通过不同策略放大稀疏潜变量以改变剪枝决策。其中，按类别引导发现紧凑的类别特异性头子集，可保持精度。例如，对于‘碗’类，准确率从76%提升至82%，头使用率由0.72降至0.33，仅依赖头h2和h5。结果表明，稀疏潜变量实现了对动态剪枝的类别级控制，有效弥合了剪枝效率与机制可解释性之间的差距。
原文摘要 · Abstract (English)
Dynamic head pruning in Vision Transformers (ViTs) improves efficiency by removing redundant attention heads, but existing pruning policies are often difficult to interpret and control. In this work, we propose a novel framework by integrating Sparse Autoencoders (SAEs) with dynamic pruning, leveraging their ability to disentangle dense embeddings into interpretable and controllable sparse latents. Specifically, we train an SAE on the final-layer residual embedding of the ViT and amplify the sparse latents with different strategies to alter pruning decisions. Among them, per-class steering reveals compact, class-specific head subsets that preserve accuracy. For example, bowl improves accuracy (76% to 82%) while reducing head usage (0.72 to 0.33) via heads h2 and h5. These results show that sparse latent features enable class-specific control of dynamic pruning, effectively bridging pruning efficiency and mechanistic interpretability in ViTs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。