无需梯度与标签，用激活空间拓扑选关键层，提升移动端模型适配效率
AdaBet: Gradient-free Layer Selection for Efficient Training of Deep Neural Networks
- 通过贝蒂数分析激活空间拓扑，仅用前向传播排序层重要性
- 在16组测试中平均比基线高2.5%准确率，峰值内存降低40%
- 适合资源受限设备的无监督在线微调，无需标注数据或反向传播
为在边缘和移动设备上高效利用预训练神经网络，需在计算与内存受限条件下，根据用户特定运行时数据分布进行适应性调整。设备端使用目标数据集重训练可实现此目的，但现代深层网络带来的计算开销使基于梯度的全层优化难以实施。现有方法虽通过选择部分层重训练降低开销，但依赖标注数据、至少一次完整模型反向传播或服务器端元训练，限制了其在受限设备上的适用性。本文提出AdaBet，一种无梯度层选择方法，通过分析激活空间的拓扑特征（贝蒂数）并仅使用前向传播对层重要性排序。该方法可在不依赖标签或梯度的情况下，选出学习能力强的关键层用于重训练与适配。在16组基准模型与数据集上的评估显示，AdaBet相比梯度基线平均提升2.5%分类准确率，同时平均峰值内存消耗降低40%。
原文摘要 · Abstract (English)
To utilize pre-trained neural networks on edge and mobile devices, we often require efficient adaptation to user-specific runtime data distributions while operating under limited compute and memory resources. On-device retraining with a target dataset can facilitate such adaptations; however, it remains impractical due to the increasing depth of modern neural nets, as well as the computational overhead associated with gradient-based optimization across all layers. Current approaches reduce training cost by selecting a subset of layers for retraining, however, they rely on labeled data, at least one full-model backpropagation, or server-side meta-training; limiting their suitability for constrained devices. We introduce AdaBet, a gradient-free layer selection approach to rank important layers by analyzing topological features of their activation spaces through Betti Numbers and using forward passes alone. AdaBet allows selecting layers with high learning capacity, which are important for retraining and adaptation, without requiring labels or gradients. Evaluating AdaBet on sixteen pairs of benchmark models and datasets, shows AdaBet achieves an average gain of 2.5% more classification accuracy over gradient-based baselines while reducing average peak memory consumption by 40%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。