模型自动扩容，学习新知识时动态增加容量，不需预设大小。
LACE: Loss-Adaptive Capacity Expansion for Continual Learning
- 根据损失信号自动判断是否扩容，不足时增加投影层维度。
- 在真实数据上达到大模型精度，起始容量仅为后者几分之一。
- 无需标签或内存，适合设备端持续学习，可识别领域切换点。
固定表示容量是持续学习中的根本限制：训练前需猜测模型宽度，却无法预知数据包含多少概念。我们提出LACE（损失自适应容量扩展），一种简单在线机制，通过监控自身损失信号在训练中扩展模型的表示容量。当持续损失偏差超过阈值——表明当前容量不足以处理新数据——LACE在投影层新增维度，并与现有参数联合训练。在合成与真实数据实验中，LACE仅在领域边界触发扩容（100%边界精度，零误报），在起始容量仅为大型固定容量模型几分之一的情况下，仍达到其精度水平；移除所有适配器导致3%准确率下降，说明这些适配器整体关键。我们进一步通过分层聚类在GPT-2激活中实现无监督领域分离，发现层间可分性呈U形曲线，为深度网络中自适应容量分配提供依据。LACE无需标签、无需回放缓冲区、无需外部控制器，适用于资源受限下的设备端持续学习。
原文摘要 · Abstract (English)
Fixed representational capacity is a fundamental constraint in continual learning: practitioners must guess an appropriate model width before training, without knowing how many distinct concepts the data contains. We propose LACE (Loss-Adaptive Capacity Expansion), a simple online mechanism that expands a model's representational capacity during training by monitoring its own loss signal. When sustained loss deviation exceeds a threshold - indicating that the current capacity is insufficient for newly encountered data - LACE adds new dimensions to the projection layer and trains them jointly with existing parameters. Across synthetic and real-data experiments, LACE triggers expansions exclusively at domain boundaries (100% boundary precision, zero false positives), matches the accuracy of a large fixed-capacity model while starting from a fraction of its dimensions, and produces adapter dimensions that are collectively critical to performance (3% accuracy drop when all adapters removed). We further demonstrate unsupervised domain separation in GPT-2 activations via layer-wise clustering, showing a U-shaped separability curve across layers that motivates adaptive capacity allocation in deep networks. LACE requires no labels, no replay buffers, and no external controllers, making it suitable for on-device continual learning under resource constraints.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。