通过早期补丁剪枝，实现视频实例分割的高效计算。
Video Patch Pruning: Efficient Video Instance Segmentation via Early Token Reduction
- 利用时序先验知识在早期网络层进行可微分补丁选择。
- 视频补丁剪枝最高达60%，远超传统图像方法30%的极限。
- 适合需要高效率视频理解的实时应用，如自动驾驶。
视觉变换器（ViTs）在多个基准上表现卓越，但其高计算开销限制了实际部署。补丁剪枝虽能显著节省资源，但现有方法仅在深层网络中进行令牌压缩，忽略了早期阶段的优化潜力。本文提出一种新型视频补丁剪枝框架（VPP），通过引入时序先验知识，在早期ViT层中实现高效的稀疏性。我们观察到深层特征具有强前景选择性，据此设计全可微模块，精准筛选早期阶段的关键补丁。该方法在密集预测任务中实现最高60%的补丁缩减，显著优于传统图像剪枝约30%的水平。在高稀疏度下，性能依然稳定，于Youtube-VIS 2021数据集上最大性能下降仅0.6%。
原文摘要 · Abstract (English)
Vision Transformers (ViTs) have demonstrated state-ofthe-art performance in several benchmarks, yet their high computational costs hinders their practical deployment. Patch Pruning offers significant savings, but existing approaches restrict token reduction to deeper layers, leaving early-stage compression unexplored. This limits their potential for holistic efficiency. In this work, we present a novel Video Patch Pruning framework (VPP) that integrates temporal prior knowledge to enable efficient sparsity within early ViT layers. Our approach is motivated by the observation that prior features extracted from deeper layers exhibit strong foreground selectivity. Therefore we propose a fully differentiable module for temporal mapping to accurately select the most relevant patches in early network stages. Notably, the proposed method enables a patch reduction of up to 60% in dense prediction tasks, exceeding the capabilities of conventional image-based patch pruning, which typically operate around a 30% patch sparsity. VPP excels the high-sparsity regime, sustaining remarkable performance even when patch usage is reduced below 55%. Specifically, it preserves stable results with a maximal performance drop of 0.6% on the Youtube-VIS 2021 dataset.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。