轻量级模型实现摄像头端实时分割，保护隐私且无需云端处理。
PicoSAM2: Low-Latency Segmentation In-Sensor for Edge Vision Applications
- 基于深度可分离U-Net与知识蒸馏，支持提示驱动分割。
- 在IMX500上仅需14.3毫秒，内存仅1.22MB，满足传感器部署要求。
- 适合智能眼镜、物联网等对延迟和隐私敏感的边缘设备。
实时、本地化分割对于智能眼镜和物联网设备等低延迟、高隐私需求的应用至关重要。我们提出PicoSAM2，一个轻量级（1.3M参数，336M MACs）的可提示分割模型，专为边缘和传感器端执行优化，包括Sony IMX500。其基于深度可分离U-Net架构，通过知识蒸馏与定点提示编码从Segment Anything Model 2（SAM2）中学习。在COCO和LVIS数据集上分别达到51.9%和44.9%的mIoU。量化后模型（1.22MB）在IMX500上运行耗时14.3毫秒，实现86 MACs/cycle的能效。知识蒸馏使LVIS上的性能提升+3.5% mIoU和+5.1% mAP。结果表明，在摄像头端实现高效可提示分割是可行的，无需依赖云端或主机处理，真正实现隐私保护视觉计算。
原文摘要 · Abstract (English)
Real-time, on-device segmentation is critical for latency-sensitive and privacy-aware applications like smart glasses and IoT devices. We introduce PicoSAM2, a lightweight (1.3M parameters, 336M MACs) promptable segmentation model optimized for edge and in-sensor execution, including the Sony IMX500. It builds on a depthwise separable U-Net, with knowledge distillation and fixed-point prompt encoding to learn from the Segment Anything Model 2 (SAM2). On COCO and LVIS, it achieves 51.9% and 44.9% mIoU, respectively. The quantized model (1.22MB) runs at 14.3 ms on the IMX500-achieving 86 MACs/cycle, making it the only model meeting both memory and compute constraints for in-sensor deployment. Distillation boosts LVIS performance by +3.5% mIoU and +5.1% mAP. These results demonstrate that efficient, promptable segmentation is feasible directly on-camera, enabling privacy-preserving vision without cloud or host processing.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。