用大模型设计更小更快的智能模型，兼顾精度与可解释性。
Can LLMs Revolutionize the Design of Explainable and Efficient TinyML Models?
- 用大模型引导搜索，结合帕累托优化，自动找最优架构。
- 三款模型在嵌入式芯片上达73%以上准确率，低于1亿次计算量。
- 适合想做高效、可解释小模型的研究者或开发者参考。
本文提出一种面向微型机器学习（TinyML）平台的新型神经网络架构设计框架。通过利用大语言模型（LLM）进行神经架构搜索（NAS），采用基于视觉变换器（ViT）的知识蒸馏（KD）策略，并集成可解释性模块，实现精度、计算效率与内存占用之间的最佳平衡。LLM引导的搜索在分层空间中探索候选架构，基于准确率、乘加操作（MACs）和内存指标进行帕累托优化。最优架构通过使用预训练ViT-B/16模型的logits知识蒸馏进一步微调，提升泛化能力而不增加模型尺寸。在CIFAR-100数据集上评估，并部署于STM32H7微控制器（MCU），提出的三个模型LMaNet-Elite、LMaNet-Core和QwNet-Core分别达到74.50%、74.20%和73.00%的准确率。所有模型均超越现有最先进（SOTA）方法，如MCUNet-in3/in4（69.62%/72.86%）和XiNet（72.27%），同时保持计算成本低于1亿次MACs，且静态随机存取内存（SRAM）约束在320 KB以内。结果表明该框架在TinyML平台上的高效性与性能优势，凸显了结合LLM驱动搜索、帕累托优化、知识蒸馏与可解释性在构建精准、高效、可解释模型方面的潜力。
原文摘要 · Abstract (English)
This paper introduces a novel framework for designing efficient neural network architectures specifically tailored to tiny machine learning (TinyML) platforms. By leveraging large language models (LLMs) for neural architecture search (NAS), a vision transformer (ViT)-based knowledge distillation (KD) strategy, and an explainability module, the approach strikes an optimal balance between accuracy, computational efficiency, and memory usage. The LLM-guided search explores a hierarchical search space, refining candidate architectures through Pareto optimization based on accuracy, multiply-accumulate operations (MACs), and memory metrics. The best-performing architectures are further fine-tuned using logits-based KD with a pre-trained ViT-B/16 model, which enhances generalization without increasing model size. Evaluated on the CIFAR-100 dataset and deployed on an STM32H7 microcontroller (MCU), the three proposed models, LMaNet-Elite, LMaNet-Core, and QwNet-Core, achieve accuracy scores of 74.50%, 74.20% and 73.00%, respectively. All three models surpass current state-of-the-art (SOTA) models, such as MCUNet-in3/in4 (69.62% / 72.86%) and XiNet (72.27%), while maintaining a low computational cost of less than 100 million MACs and adhering to the stringent 320 KB static random-access memory (SRAM) constraint. These results demonstrate the efficiency and performance of the proposed framework for TinyML platforms, underscoring the potential of combining LLM-driven search, Pareto optimization, KD, and explainability to develop accurate, efficient, and interpretable models. This approach opens new possibilities in NAS, enabling the design of efficient architectures specifically suited for TinyML.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。