为超小模型设计自监督预训练，让396K参数的MCU模型性能大幅提升
TinySSL: Distilled Self-Supervised Pretraining for Sub-Megabyte MCU Models
- 用教师模型引导的非对称蒸馏，解决小模型表示瓶颈问题
- 在CIFAR-100上线性探测准确率达62.7%，比基线高18个百分点
- 适合资源受限设备部署，仅占378KB内存且无推理开销
自监督学习（SSL）已革新大模型表征学习，但在参数少于500K的微控制器（MCU）模型中仍鲜有探索。我们识别出三个关键障碍：投影头主导、表征瓶颈和增强敏感性，并提出容量感知蒸馏自监督学习（CA-DSSL），一种无需标签或文本监督的教师引导框架。CA-DSSL结合冻结的DINO ViT-S/16教师模型进行非对称蒸馏、多尺度特征蒸馏以保留空间信息，以及渐进式增强课程。在MobileNetV2-0.35主干网络（396K参数）上预训练的CIFAR-100数据集上，CA-DSSL达到62.7±0.5%的线性探测准确率（三种子均值），较SimCLR-Tiny提升18个百分点，与SEED（61.7%）相当但仅需426K vs. 3.15M的投影参数，达到监督上限的94.0%。标准SSL方法（BYOL-Tiny, DINO-Tiny）在此规模下完全崩溃。在Pascal VOC检测任务中，CA-DSSL实现随机初始化2.3倍的mAP，较SEED提升3个百分点，尽管SimCLR-Tiny在检测mAP上与之持平。部署后主干模型仅占378 KB（INT8），预训练无推理开销。初步ImageNet-100实验表明，CA-DSSL的优势集中在小数据场景；扩展至ImageNet-1K将作为未来工作。
原文摘要 · Abstract (English)
Self-supervised learning (SSL) has transformed representation learning for large models, yet remains unexplored for microcontroller (MCU)-class models with fewer than 500K parameters. We identify three obstacles at this scale -- projection head dominance, representation bottleneck, and augmentation sensitivity -- and propose Capacity-Aware Distilled Self-Supervised Learning (CA-DSSL), a teacher-guided framework that overcomes them without labels or text supervision. CA-DSSL combines asymmetric distillation from a frozen DINO ViT-S/16 teacher, multi-scale feature distillation for spatial representations, and a progressive augmentation curriculum. On a MobileNetV2-0.35 backbone (396K parameters) pretrained on CIFAR-100, CA-DSSL reaches 62.7 0.5% linear-probe accuracy (3-seed mean) -- surpassing SimCLR-Tiny by 18 pp, matching SEED (61.7%) with 10 fewer projection parameters (426K vs. 3.15M), and reaching 94.0% of a supervised upper bound. Standard SSL methods (BYOL-Tiny, DINO-Tiny) collapse entirely at this scale. On Pascal VOC detection, CA-DSSL achieves 2.3 the mAP of random initialization and +3 pp over SEED, though SimCLR-Tiny matches CA-DSSL on detection mAP. The deployed backbone occupies 378 KB (INT8) with no inference overhead from pretraining. Preliminary ImageNet-100 experiments reveal that CA-DSSL's advantage is specific to small-data regimes; scaling to ImageNet-1K is discussed as future work.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。