自动调整编码器隐藏层大小，用更少计算量实现同等压缩效果
SOSAE: Self-Organizing Sparse AutoEncoder
- 基于物理启发的正则化机制，动态调节特征空间维度
- 在多个数据集上节省130倍浮点运算量，性能不变
- 适合追求高效模型压缩的工程应用
调整自编码器隐藏层尺寸可获得输入数据的最佳压缩表示，但传统网格搜索方式耗时耗算力。本文提出自组织稀疏自编码器（SOSAE），通过引入结构化稀疏性正则化，使特征空间维度在训练中动态收缩。该方法基于特征向量分量的大小和位置索引进行惩罚，促使非活跃部分被截断而无信息损失。大量实验表明，SOSAE相比其他基线，在保持相同调优质量与性能的前提下，将浮点运算量（FLOPs）减少高达130倍。
原文摘要 · Abstract (English)
The process of tuning the size of the hidden layers for autoencoders has the benefit of providing optimally compressed representations for the input data. However, such hyper-parameter tuning process would take a lot of computation and time effort with grid search as the default option. In this paper, we introduce the Self-Organization Regularization for Autoencoders that dynamically adapts the dimensionality of the feature space to the optimal size. Inspired by physics concepts, Self-Organizing Sparse AutoEncoder (SOSAE) induces sparsity in feature space in a structured way that permits the truncation of the non-active part of the feature vector without any loss of information. This is done by penalizing the autoencoder based on the magnitude and the positional index of the feature vector dimensions, which during training constricts the feature space in both terms. Extensive experiments on various datasets show that our SOSAE can tune the feature space dimensionality up to 130 times lesser Floating-point Operations (FLOPs) than other baselines while maintaining the same quality of tuning and performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。