评测了六种元持续学习方法在边缘设备上的表现,兼顾准确率与资源消耗。
MetaCLBench: Meta Continual Learning Benchmark on Resource-Constrained Edge Devices
- 构建真实物联网设备上的基准测试框架,评估模型部署可行性。
- 三类模型在1GB以下设备上均出现内存溢出,限制实际应用。
- 提出实用部署建议,支持公平对比性能与系统指标。
元持续学习(Meta-CL)使模型能在少量标注样本下学习新类别,适用于人工标注成本高的物联网场景。然而现有研究多关注准确率,忽视资源受限硬件的部署可行性。为此,我们提出MetaCLBench,一个在真实物联网设备上评估Meta-CL方法准确率与关键系统指标(内存占用、延迟、能耗)的基准框架,设备内存范围为512 MB至4 GB。我们在三种架构(CNN、YAMNet、ViT)和五个数据集(涵盖图像与音频模态)上评估了六种Meta-CL方法。结果显示,根据数据集不同,最多有三种方法在小于1 GB内存设备上发生内存溢出,显著缩小可行部署选项。LifeLearner在保持接近最优准确率的同时,能耗比最优基线低2.54–7.43倍。值得注意的是,更大型或复杂的模型如ViT和YAMNet并不一定带来更好性能,结果因数据集和模态而异,挑战了模型复杂度决定效果的固有认知。最后,我们提供实用部署指南,并将在发表后公开框架,以促进准确率与系统级指标的公平评估。
原文摘要 · Abstract (English)
Meta-Continual Learning (Meta-CL) enables models to learn new classes from limited labelled samples, making it promising for IoT applications where manual labelling is costly. However, existing studies focus on accuracy while ignoring deployment viability on resource-constrained hardware. Thus, we present MetaCLBench, a benchmark framework that evaluates Meta-CL methods for both accuracy and deployment-critical metrics (memory footprint, latency, and energy consumption) on real IoT devices with RAM sizes ranging from 512 MB to 4 GB. We evaluate six Meta-CL methods across three architectures (CNN, YAMNet, ViT) and five datasets spanning image and audio modalities. Our evaluation reveals that, depending on the dataset, up to three of six methods cause out-of-memory failures on sub-1 GB devices, significantly narrowing viable deployment options. LifeLearner achieves near-oracle accuracy while consuming 2.54-7.43x less energy than the Oracle method. Notably, larger or more sophisticated architectures such as ViT and YAMNet do not necessarily yield better Meta-CL performance, with results varying across datasets and modalities, challenging conventional assumptions about model complexity. Finally, we provide practical deployment guidelines and will release our framework upon publication to enable fair evaluation across both accuracy and system-level metrics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。