轻量编码器让传感器端高效处理图像,大幅降低传输能耗。
OASIS: Optimized Lightweight Autoencoder System for Distributed In-Sensor computing
- 传感器端部署轻量双分支自编码器,压缩图像特征
- 输出维度降低四数量级,能效提升2-4.5倍
- 适合智能家庭与增强现实等低功耗视觉场景
在传感器内计算通过在芯片上直接处理数据,缓解高分辨率、高帧率视频传输带来的带宽瓶颈,尤其适用于AR/VR和智能家居系统。我们提出一种将CMOS图像传感器(CIS)与逻辑芯片通过先进封装集成的架构,由逻辑芯片运行深度神经网络(DNN)的前几层。然而,受限于算力和内存,部署复杂DNN仍具挑战。传统模型分割方式因前期需多层处理,难以实现显著压缩。为此,我们设计了一种基于双分支自编码器的视觉架构:在逻辑芯片上部署轻量编码器,任务相关网络保留在外部。编码器采用三重损失函数训练:(1)任务特定损失提升精度,(2)熵损失促进紧凑可压缩表示,(3)重建损失(均方误差)保留关键视觉信息。该设计使输出激活维度相比输入图像降低四个数量级,经硬件支持的半解析能效模型验证,能量消耗降低2–4.5倍。我们在智能家庭与增强现实场景中评估了基于CNN和ViT的模型,达到当前最优精度,能效高达22.7 TOPS/W。
原文摘要 · Abstract (English)
In-sensor computing, which integrates computation directly within the sensor, has emerged as a promising paradigm for machine vision applications such as AR/VR and smart home systems. By processing data on-chip before transmission, it alleviates the bandwidth bottleneck caused by high-resolution, high-frame-rate image transmission, particularly in video applications. We envision a system architecture that integrates a CMOS image sensor (CIS) with a logic chip via advanced packaging, where the logic chip processes early-stage deep neural network (DNN) layers. However, its limited compute and memory make deploying advanced DNNs challenging. A simple solution is to split the model, executing the first part on the logic chip and the rest off-chip. However, modern DNNs require multiple layers before dimensionality reduction, limiting their ability to achieve the primary goal of in-sensor computing: minimizing data bandwidth. To address this, we propose a dual-branch autoencoder-based vision architecture that deploys a lightweight encoder on the logic chip while the task-specific network runs off-chip. The encoder is trained using a triple loss function: (1) task-specific loss to optimize accuracy, (2) entropy loss to enforce compact and compressible representations, and (3) reconstruction loss (mean-square error) to preserve essential visual information. This design enables a four-order-of-magnitude reduction in output activation dimensionality compared to input images, resulting in a $2{-}4.5\times$ decrease in energy consumption, as validated by our hardware-backed semi-analytical energy models. We evaluate our approach on CNN and ViT-based models across applications in smart home and augmented reality domains, achieving state-of-the-art accuracy with energy efficiency of up to 22.7 TOPS/W.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。