用注意力机制设计记忆模块,让模型在低功耗下持续学习数据流。
An Attention-based Feature Memory Design for Energy-Efficient Continual Learning
- 用注意力机制动态存储和召回关键特征,减少重复训练。
- 在概念漂移数据上,能耗降低最高达33.3%,精度仅下降0.062。
- 适合边缘设备部署,尤其看重能效比的实时决策场景。
表格式数据流在医疗、金融和物联网等实时决策场景中日益普遍,通常在资源受限的边缘和移动设备上生成与处理。持续学习(CL)使模型能够顺序学习此类数据流并保留已有知识。尽管近期CL研究显著缓解了灾难性遗忘问题,但针对表格式数据流的持续学习在能效与内存效率方面仍缺乏探索。为此,本文提出AttenMLP,结合注意力机制的特征重放、上下文检索与滑动缓冲更新,在小批量训练框架下实现流式表格式学习。我们在真实世界概念漂移基准上评估AttenMLP,对比SOTA表格式模型。实验表明,AttenMLP在无需重放的情况下达到与强基线相当的准确率,同时通过可调设计显著降低能耗。具体而言,在增量概念漂移数据集上,精度仅下降0.062,能耗降低最高达33.3%(相较TabPFNv2);在突发概念漂移数据集上,能耗较TabR降低1.47%,精度下降0.038。虽全局效率排名第三,但其在两类概念漂移场景中均展现出优于现有表格式模型的能效-精度权衡能力。
原文摘要 · Abstract (English)
Tabular data streams are increasingly prevalent in real-time decision-making across healthcare, finance, and the Internet of Things, often generated and processed on resource-constrained edge and mobile devices. Continual learning (CL) enables models to learn sequentially from such streams while retaining previously acquired knowledge. While recent CL advances have made significant progress in mitigating catastrophic forgetting, the energy and memory efficiency of CL for tabular data streams remains largely unexplored. To address this gap, we propose AttenMLP, which integrates attention-based feature replay with context retrieval and sliding buffer updates within a minibatch training framework for streaming tabular learning. We evaluate AttenMLP against state-of-the-art (SOTA) tabular models on real-world concept drift benchmarks with temporal distribution shifts. Experimental results show that AttenMLP achieves accuracy comparable to strong baselines without replay, while substantially reducing energy consumption through tunable design choices. In particular, with the proposed attention-based feature memory design, AttenMLP costs a 0.062 decrease in final accuracy under the incremental concept drift dataset, while reducing energy usage up to 33.3\% compared to TabPFNv2. Under the abrupt concept drift dataset, AttenMLP reduces 1.47\% energy consumption compared to TabR, at the cost of a 0.038 decrease in final accuracy. Although ranking third in global efficiency, AttenMLP demonstrates energy-accuracy trade-offs across both abrupt and incremental concept drift scenarios compared to SOTA tabular models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。