arXiv:2605.21171cs.CV2026-05

全量化视觉Transformer,模型压缩15倍,可在微控制器上运行。

FTerViT: Fully Ternary Vision Transformer

论文配图:FTerViT: Fully Ternary Vision Transformer
图 1 · 摘自论文原文
  • 所有层参数与归一化参数均转为三值,实现全量三值化。
  • 在384×384分辨率下达82.43%精度,仅占6.09MB内存。
  • 首次在ESP32-S3微控制器上部署三值视觉模型。

三值视觉Transformer虽可大幅压缩模型,但现有方法仅对编码器层进行三值化,仍保留分块嵌入、归一化参数和分类头的全精度,限制了资源受限设备(如微控制器)上的部署效率。本文提出全三值化视觉Transformer(FTerViT),首次实现所有权重矩阵与归一化参数的三值化。为此,设计了支持通道级缩放的三值卷积算子(TernaryBitConv2d)与三值归一化(TernaryLayerNorm)。FTerViT通过知识蒸馏训练,并辅以轻量级量化感知恢复阶段。在384×384分辨率下,三值W2A8 DeiT-III-S达到82.43% ImageNet-1K Top-1精度,模型大小仅6.09MB(约15倍压缩,相比全精度损失2.42个百分点),优于此前三值方法最高8个百分点。最后,我们首次在ESP32-S3芯片内的双核XTensa LX7微控制器上部署了FTerViT-Small(基于DeiT-III-Small,224×224,5.81MB),实现79.64%精度。

原文摘要 · Abstract (English)

Ternary Vision Transformers offer substantial model compression, however state-of-the-art methods only ternarize the encoder layers, leaving patch embeddings, LayerNorm parameters, and classifier heads in full precision. In compact models targeting resource-constrained processors, such as microcontrollers, these remaining full-precision components determine the total memory footprint, severely limiting deployment efficiency and on-device feasibility. In this work, we introduce a fully ternarized Vision Transformer in which \emph{all} weight matrices and normalization parameters are ternarized (FTerViT). To this end, we introduce two novel operators : TernaryBitConv2d with per-channel scaling for patch embedding and TernaryLayerNorm. FTerViT is trained using knowledge distillation, followed by a lightweight quantization-aware recovery phase. Our ternary W2A8 DeiT-III-S at 384$\times$384 resolution achieves 82.43\% ImageNet-1K top-1 at 6.09\,MB (${\sim}$15$\times$ compression, $-$2.42\,pp vs.\ FP32), outperforming prior ternary ViTs methods up to 8 pp. Finally, we demonstrate the first implementation of ternary vision transformers on a dual cores XTensa LX7 microcontroller inside the ESP32-S3 system-on-chip. By deploying FTerViT-Small (based on DeiT-III-Small at 224$\times$224 resolution, 5.81\,MB), we achieve 79.64\% ImageNet-1K top-1 accuracy.

视觉Transformer模型压缩三值化边缘计算

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。