为视觉Transformer设计可适配任务的剪枝策略，提升多任务效率。
Putting Registers to Work: Task Registers for Token Pruning in Vision Transformers
- 引入任务寄存器动态管理剪枝，按任务激活对应寄存器
- 在ADE20K上达47.0 mIoU，COCO上53.7 box AP，吞吐提升1.3倍
- 适用于分类、分割、检测等多任务场景，尤其适合资源受限部署
现有令牌剪枝策略通常针对单一识别流程设计，但预训练视觉Transformer需在图像分类、语义分割和目标检测等不同任务间复用。本文通过受控探针实验发现：分割与检测对剪枝标准排序不同，分类对早期层注意力剪枝更敏感，密集任务偏好相反的恢复终点。据此提出任务自适应剪枝（TAP），为每类任务配置独立任务寄存器，仅激活当前任务寄存器。其状态演化用于排序令牌，分配精确剪枝预算至深度，并设置密集特征恢复尺度。在保持保留率ρ=0.5时，联合适配模型TAP-J在ADE20K上实现47.0 mIoU，编码器吞吐达1.30×；在COCO上达53.7 box AP，吞吐1.32×，同时在ImageNet-1K上保持竞争力。
原文摘要 · Abstract (English)
Token-pruning policies are usually designed for a single recognition pipeline, but pretrained Vision Transformers are reused across tasks with different spatial demands. We ask which parts of a pruning policy transfer across image classification, semantic segmentation, and object detection. For each pipeline, controlled probes freeze the no-pruning checkpoint and apply a series of parameter-free reduction criteria at one eligible layer at a time without retraining. The probes reveal three differences: segmentation and detection rank the criteria differently, classification is especially sensitive to attention-based pruning in the earliest layers, and the dense tasks prefer opposite recovery endpoints. These findings motivate Task-Adaptive Pruning (TAP). Existing register tokens serve as task-agnostic storage for feature artifacts. TAP instead introduces one task register per task and activates only the current one. Its evolving state ranks tokens, distributes an exact removal budget over depth, and sets the recovery scale for dense features. At a final keep rate of $ρ=0.5$, our jointly adapted model, TAP-J, reaches $47.0$ mIoU at $1.30\times$ encoder throughput on ADE20K and $53.7$ box AP at $1.32\times$ encoder throughput on COCO while remaining competitive on ImageNet-1K.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。