用智能搜索动态扩容模型,高效应对持续学习中的遗忘问题。
SEAL: Searching Expandable Architectures for Incremental Learning
- 根据容量评估动态决定是否扩展模型结构。
- 在多个基准上显著降低遗忘率,提升准确率。
- 适合资源受限场景下的持续学习任务。
增量学习要求模型从顺序到来的任务流中持续学习,核心挑战在于平衡可塑性(学习新任务)与稳定性(保留旧知识)。现有基于神经架构搜索(NAS)的方法通常在每个任务后都扩展模型,导致资源消耗过大。本文提出SEAL框架,专为数据增量学习设计——即不存储历史数据,仅接收连续的独立数据样本。SEAL通过容量估计指标动态决定是否扩展模型,并在每次扩展后采用交叉蒸馏训练保持稳定性。其NAS组件联合搜索最优架构与扩展策略。实验表明,SEAL在多个基准上有效减少遗忘、提升准确率,且仅在必要时增加容量。结果验证了结合NAS与选择性扩展在高效自适应学习中的潜力。
原文摘要 · Abstract (English)
Incremental learning is a machine learning paradigm where a model learns from a sequential stream of tasks. This setting poses a key challenge: balancing plasticity (learning new tasks) and stability (preserving past knowledge). Neural Architecture Search (NAS), a branch of AutoML, automates the design of the architecture of Deep Neural Networks and has shown success in static settings. However, existing NAS-based approaches to incremental learning often rely on expanding the model at every task, making them impractical in resource-constrained environments. In this work, we introduce SEAL, a NAS-based framework tailored for data-incremental learning, a scenario where disjoint data samples arrive sequentially and are not stored for future access. SEAL adapts the model structure dynamically by expanding it only when necessary, based on a capacity estimation metric. Stability is preserved through cross-distillation training after each expansion step. The NAS component jointly searches for both the architecture and the optimal expansion policy. Experiments across multiple benchmarks demonstrate that SEAL effectively reduces forgetting and enhances accuracy while allocating additional capacity only when required. These results highlight the promise of combining NAS and selective expansion for efficient, adaptive learning in incremental scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。