用视觉词汇表+类别统计,智能选图块,压缩模型不丢精度。
REDI: Corpus Aware Patch Ranking for DINOv3 Token Reduction

- 基于视觉词袋和监督TF-IDF生成图像块重要性评分
- 序列长度从201减至107,减少46.8%,准确率达84.706%
- 适合追求高效视觉模型的开发者,尤其适配DINOv3
大多数视觉变换器的标记压缩方法通过剪枝、合并或池化图像块来平衡精度与效率。REDI(DINOv3标记压缩的相关性)通过受控的监督参考,研究在固定标记预算下如何分配各图像块的资源。它将DINOv3最后一层的图像块表示量化为视觉词汇,并利用监督TF-IDF计算类别相关的语料库得分。对每个验证图像,真实类别选取TF-IDF表中的一行,四个变换视图生成与参考中心裁剪对齐的TF-IDF图。同时,在同一裁剪区域进行独立密集前向传播获得注意力图。经独立最小-最大归一化后,两图逐元素相乘得到REDI分数。固定保留、合并与压缩操作根据分数排序分配块角色,按分数大小加权合并与压缩。使用预计算的REDI分数,冻结的DINOv3 ViT-B/16主干网络及相同的线性分类器,将序列长度从201降至107,减少46.8%。基于输入注意力质量的REDI变体在ImageNet-1K上达到84.706%的Top-1准确率,优于密集基线(83.514%)、仅输入注意力(82.634%)和仅监督TF-IDF(81.796%)。相同语料库项也使三种替代注意力形式的压缩分类性能优于仅注意力版本。这些对照实验表明,类别特定语料库统计与图像特定注意力在该设定下提供互补信号。
原文摘要 · Abstract (English)
Most token reduction methods for Vision Transformers seek favorable tradeoffs between accuracy and efficiency by pruning, merging, or pooling patch tokens. REDI (Relevance for DINOv3 Token Reduction) studies this question through a controlled supervised reference: how should a fixed token budget be allocated across patches for image classification? REDI quantizes final block DINOv3 patch representations into a visual vocabulary and derives class conditioned corpus scores using supervised TF-IDF over visual words. For each validation image, the ground truth class selects a row of the TF-IDF table, and four transformed views produce a TF-IDF map aligned to a reference center crop. A separate dense pass on the same crop provides an attention map. After independent min max normalization, their elementwise product defines the REDI score. A fixed keep, merge, and compress operator then uses score rank to assign patch roles and score magnitude to weight merging and compression. With precomputed REDI scores, a frozen DINOv3 ViT-B/16 backbone, and the same linear classifier used for dense evaluation, the operator reduces the sequence length from 201 to 107 tokens, a 46.8% sequence reduction. The REDI variant based on incoming attention mass achieves 84.706% Top-1 accuracy on ImageNet-1K, compared with 83.514% for the dense baseline, 82.634% for incoming attention mass alone, and 81.796% for supervised TF-IDF alone. The same corpus term also improves reduced classification for three alternative attention formulations relative to their attention only counterparts. Together, these controlled comparisons indicate that class specific corpus statistics and image specific attention provide complementary signals for patch ranking in this setting.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。