TinyVLM让微控制器实现零样本目标检测，仅用不到1MB内存。
TinyVLM: Zero-Shot Object Detection on Microcontrollers via Vision-Language Distillation with Matryoshka Embeddings
- 分离视觉与文本处理，预存类别嵌入节省内存
- 多层嵌入蒸馏实现精度与内存灵活权衡
- 量化存储使原型内存减少4倍，适合边缘设备
零样本目标检测可识别未训练过的物体，但现有方法依赖如CLIP等大型视觉语言模型，需数百兆内存，远超微控制器（MCU）限制。本文提出TinyVLM，首个在资源受限MCU上实现零样本检测的框架，内存低于1MB。创新包括：(1) 解耦架构，将视觉推理与文本编码分离，使类别嵌入可预存于闪存；(2) 马特里什卡蒸馏，在16-256维多层嵌入中训练，支持灵活的精度-内存权衡；(3) 量化嵌入存储，使类别原型内存减少4倍且精度损失极小。在CC3M上训练，TinyVLM在COCO、Flowers102和Food101上表现良好，部署后仅需285KB RAM和892KB闪存。在STM32H7上实现实时推理（26 FPS），MAX78000上超过1,000 FPS，首次实现边缘设备上的实用零样本检测。
原文摘要 · Abstract (English)
Zero-shot object detection enables recognising novel objects without task-specific training, but current approaches rely on large vision language models (VLMs) like CLIP that require hundreds of megabytes of memory - far exceeding the constraints of micro controller units (MCUs). We present TinyVLM, the first framework enabling zero-shot object detection on resource-constrained MCUs with less than 1MB of memory. Our approach introduces three key innovations: (1) a decoupled architecture that separates visual inference from text encoding, allowing precomputed class embeddings to be stored in flash memory; (2) Matryoshka distillation that trains nested embeddings at multiple dimensions (16-256), enabling flexible accuracy-memory trade-offs; and (3) quantized embedding storage that reduces class prototype memory by 4x with minimal accuracy loss. Trained on Conceptual Captions 3M (CC3M), TinyVLM achieves competitive zero-shot accuracy on COCO, Flowers102, and Food101 while requiring only 285KB of RAM and 892KB of flash memory for the deployed vision encoder. We demonstrate real-time inference at 26 FPS on STM32H7 and over 1,000 FPS on MAX78000 with its CNN accelerator, enabling practical zero-shot detection on edge devices for the first time.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。