arXiv:2608.08832cs.CV2026-08

提出VTC编码器,高效压缩ViT中间特征,节省37倍带宽。

Visual Token Codec: Unleashing Spatial Redundancy for ViT Feature Coding

论文配图:Visual Token Codec: Unleashing Spatial Redundancy for ViT Feature Coding
图 1 · 摘自论文原文
  • 分路编码全局与局部令牌,保留图像网格结构
  • 在90%性能下压缩率提升15.7至37.4倍
  • 适合分布式视觉模型部署与存储优化

大型视觉基础模型的分布式部署常需在计算节点间传输中间令牌特征,高效压缩至关重要。现有方法将异构的全局与块级令牌展平为伪图像,导致熵模型仅捕捉序列依赖,忽略原始二维块网格结构。本文发现ViT块令牌在原始网格上仍具强局部空间相关性。为此提出视觉令牌编码器(VTC),采用双路径学习编码:全局令牌用轻量因子化先验压缩,块令牌则在块-令牌网格上使用空间-通道上下文熵模型编码。为支持中间层压缩与可变码率,VTC引入后续ViT模块后的特征匹配监督及单编码器内可变码率模块。在DINOv2和SAM3上的实验表明,VTC在分类、分割与检测任务中持续优于主流基线。在保持90%未压缩特征性能下,比特率降低15.7倍至37.4倍。进一步提供了面向传输与存储的中间层率-效分析。

原文摘要 · Abstract (English)

Distributed deployment of large vision foundation models often partitions a ViT backbone and exchanges intermediate token features between computing nodes, making efficient feature compression critical under bandwidth and computation constraints. Existing ViT feature codecs typically flatten heterogeneous global and patch tokens into an L x C pseudo image, causing entropy models to mainly capture sequence-axis dependencies while overlooking the native two-dimensional patch-grid structure. In this paper, we show that ViT patch tokens retain strong local spatial correlations on the original grid. To exploit this structural prior, we propose the Visual Token Codec (VTC), a dual-path learned codec that separates global and patch tokens into dedicated coding paths. Global tokens are compressed with a lightweight factorized prior, whereas patch tokens are encoded on the patch-token grid using a spatial-channel context entropy model. To support intermediate-layer compression and practical rate adaptation, VTC further incorporates feature-matching supervision after subsequent ViT blocks and variable-rate modules within a single codec. Experiments on DINOv2 and SAM3 show that VTC consistently outperforms representative ViT feature coding baselines on classification, segmentation, and detection tasks. At 90% of uncompressed-feature performance, VTC reduces bitrate by 15.7x-37.4x across these tasks. We further provide intermediate-layer rate-utility analyses for practical transmission- and storage-oriented deployment scenarios.

ViT编码特征压缩分布式训练空间冗余

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。