arXiv:2606.19574eess.IVcs.CV2026-06

通过频域压缩图像数据,显著降低边缘视觉模型的传输能耗。

FrequencyFormer: A Co-Designed Sensor-to-Processor Pipeline for Frequency-Domain Vision Transformer Inference

论文配图:FrequencyFormer: A Co-Designed Sensor-to-Processor Pipeline for Frequency-Domain Vision Transformer Inference
图 1 · 摘自论文原文
  • 在传感器端用DCT将图像转为频域特征,实现128倍数据压缩
  • 硬件采用查表法无乘法计算,能效与面积效率双提升
  • 兼容预训练模型,适合部署于资源受限的边缘设备

将视觉变换器(ViTs)部署在传感器边缘系统中,不仅受限于本地计算能力,还受高维图像数据从传感器到处理器传输所消耗的能源和带宽制约。现有近传感器计算方法虽能部分减少数据量,但压缩效果有限。我们观察到频域天然具备紧凑的视觉信息表达能力,可在传感器端直接利用以减少数据移动。基于此,提出FrequencyFormer,一种协同设计的传感器-处理器流水线,用于高效ViT推理。该方案包括:(1) 多尺度DCT标记器,将224x224图像压缩为紧凑的频域标记,实现高达128倍的片外数据量减少,精度损失微小;(2) 基于查找表的近传感器硬件实现,利用固定DCT系数实现无乘法、低功耗、低面积的标记化;(3) 改进的基于MIPI的低功耗通信架构,进一步降低传输能耗。FrequencyFormer可作为标准ViT块嵌入的即插即用替代方案,兼容分类、检测和分割任务中的预训练主干网络。该系统达到28.8 TOPS/W的能效,通信能耗降低230倍,传感器侧总能耗降低2.22倍,证明频域标记化是传感器级ViT部署的可扩展基础。

原文摘要 · Abstract (English)

Deploying vision transformers (ViTs) on sensor-edge systems is limited not only by on-device compute, but also by the energy and bandwidth required to transmit high-dimensional image data from the sensor to the processor. While in-sensor and near-sensor computing reduce this cost through early feature extraction, existing methods often provide only modest compression. We observe that the frequency domain provides a naturally compact representation of visual information and can be exploited at the sensor level to reduce sensor-to-processor data movement. Building on this insight, we present FrequencyFormer, a co-designed sensor-to-processor pipeline for efficient ViT inference. FrequencyFormer includes: (1) a multi-scale DCT tokenizer that compresses a 224x224 image into compact frequency-domain tokens, achieving up to 128x reduction in off-chip data volume with modest accuracy loss; (2) a LUT-based near-sensor hardware implementation that leverages fixed DCT coefficients for multiplier-free, energy- and area-efficient tokenization; and (3) a modified MIPI-based low-power communication architecture that further reduces transfer energy. FrequencyFormer serves as a drop-in replacement for standard ViT patch embedding and remains compatible with pretrained backbones across classification, detection, and segmentation tasks. The pipeline achieves 28.8 TOPS/W, reduces communication energy by 230x, and lowers total sensor-side energy by 2.22x, demonstrating frequency-domain tokenization as a scalable foundation for in-sensor ViT deployment.

边缘计算视觉变换器频域处理低功耗

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。