通过交替训练稀疏与密集激活，提升模型泛化能力。
Joint Training Across Multiple Activation Sparsity Regimes
- 在不同稀疏度间循环训练，动态调整激活保留比例
- 在CIFAR-10上优于无数据增强的稠密基线
- 适合追求泛化性能的轻量化模型研究者
深度神经网络的泛化能力仍不完全清楚。受生物系统更强泛化特性的启发，我们提出假设：鲁棒的内部表征应在密集和稀疏激活状态下均有效。为此，我们引入一种简单训练策略，对隐藏层激活施加全局 top-k 约束，并通过渐进压缩与周期重置，让单个模型在多个激活预算间反复切换。使用无数据增强的 CIFAR-10 和 WRN-28-4 骨干网络，在单次运行实验中，两种自适应保留率控制策略均优于稠密基线训练。初步结果表明，跨多激活稀疏度联合训练可能为提升泛化提供一种简单而有效的方法。
原文摘要 · Abstract (English)
Generalization in deep neural networks remains only partially understood. Inspired by the stronger generalization tendency of biological systems, we explore the hypothesis that robust internal representations should remain effective across both dense and sparse activation regimes. To test this idea, we introduce a simple training strategy that applies global top-k constraints to hidden activations and repeatedly cycles a single model through multiple activation budgets via progressive compression and periodic reset. Using CIFAR-10 without data augmentation and a WRN-28-4 backbone, we find in single-run experiments that two adaptive keep-ratio control strategies both outperform dense baseline training. These preliminary results suggest that joint training across multiple activation sparsity regimes may provide a simple and effective route to improved generalization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。