用二阶敏感度动态剪枝,让ViT更快更省电。
HEART-VIT: Hessian-Guided Efficient Dynamic Attention and Token Pruning in Vision Transformer
- 基于海森向量积计算注意力头和标记的曲率敏感度,实现精准剪枝
- 在ImageNet上最多减少49.4%计算量,延迟降低36%,吞吐提升46%
- 首次统一的二阶自适应剪枝框架,适合边缘设备部署
视觉变压器(ViTs)虽精度领先,但其二次复杂度的注意力机制和冗余计算严重限制了在低延迟、低资源平台的部署。现有剪枝方法孤立处理标记或注意力头,依赖启发式或一阶信号,常牺牲精度或泛化性差。本文提出HEART-ViT,首个统一、二阶、输入自适应的ViT优化框架。通过高效海森-向量积估计标记与注意力头的曲率加权敏感度,在明确损失预算下做出合理剪枝决策。双视角敏感度分析揭示:标记剪枝主导算力节省,头剪枝实现细粒度冗余消除,二者结合获得更优平衡。在ImageNet-100与ImageNet-1K上,使用ViT-B/16与DeiT-B/16,HEART-ViT最多减少49.4% FLOPs,延迟降低36%,吞吐提升46%,微调后精度一致甚至超越基线,例如在40%标记剪枝下恢复4.7%精度。在AGX Orin等边缘设备部署验证,FLOPs与延迟降低直接转化为实际推理速度与能效提升。该框架首次实现理论与实践融合,提供兼具精度保持与边缘高效的统一剪枝方案。
原文摘要 · Abstract (English)
Vision Transformers (ViTs) deliver state-of-the-art accuracy but their quadratic attention cost and redundant computations severely hinder deployment on latency and resource-constrained platforms. Existing pruning approaches treat either tokens or heads in isolation, relying on heuristics or first-order signals, which often sacrifice accuracy or fail to generalize across inputs. We introduce HEART-ViT, a Hessian-guided efficient dynamic attention and token pruning framework for vision transformers, which to the best of our knowledge is the first unified, second-order, input-adaptive framework for ViT optimization. HEART-ViT estimates curvature-weighted sensitivities of both tokens and attention heads using efficient Hessian-vector products, enabling principled pruning decisions under explicit loss budgets.This dual-view sensitivity reveals an important structural insight: token pruning dominates computational savings, while head pruning provides fine-grained redundancy removal, and their combination achieves a superior trade-off. On ImageNet-100 and ImageNet-1K with ViT-B/16 and DeiT-B/16, HEART-ViT achieves up to 49.4 percent FLOPs reduction, 36 percent lower latency, and 46 percent higher throughput, while consistently matching or even surpassing baseline accuracy after fine-tuning, for example 4.7 percent recovery at 40 percent token pruning. Beyond theoretical benchmarks, we deploy HEART-ViT on different edge devices such as AGX Orin, demonstrating that our reductions in FLOPs and latency translate directly into real-world gains in inference speed and energy efficiency. HEART-ViT bridges the gap between theory and practice, delivering the first unified, curvature-driven pruning framework that is both accuracy-preserving and edge-efficient.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。