用预定义向量配置实现任意类别数的神经网络训练
Using predefined vector systems as latent space configuration for neural network supervised training on data with arbitrarily large number of classes
- 用预设向量系统作为目标潜在空间，解耦模型参数与类别数
- 在128万类数据上成功训练ViT，验证方法可扩展性
- 适合大规模分类、持续学习及模型蒸馏场景
监督学习（SL）是神经网络分类任务训练的核心方法，但传统方式需使模型参数随类别数增长，难以应对超大规模或未知类别数的数据。本文提出一种新方法：在训练中使用预定义向量系统作为目标潜在空间配置（LSC），使同一神经网络架构可适应任意类别数。我们采用An根系的随机扰动向量作为目标配置，在低维与高维情况下成功训练了编码器和视觉变换器（ViT），在Cinic-10和ImageNet-1K上实现准确匹配。进一步，我们在含128万类的大规模数据集上训练了ViT，验证了方法在极端类别数下的适用性。此外，文中还讨论了该方法在持续学习和模型蒸馏中的潜力，展现其广泛适用性。
原文摘要 · Abstract (English)
Supervised learning (SL) methods are indispensable for neural network (NN) training used to perform classification tasks. While resulting in very high accuracy, SL training often requires making NN parameter number dependent on the number of classes, limiting their applicability when the number of classes is extremely large or unknown in advance. In this paper we propose a methodology that allows one to train the same NN architecture regardless of the number of classes. This is achieved by using predefined vector systems as the target latent space configuration (LSC) during NN training. We discuss the desired properties of target configurations and choose randomly perturbed vectors of An root system for our experiments. These vectors are used to successfully train encoders and visual transformers (ViT) on Cinic-10 and ImageNet-1K in low- and high-dimensional cases by matching NN predictions with the predefined vectors. Finally, ViT is trained on a dataset with 1.28 million classes illustrating the applicability of the method to training on datasets with extremely large number of classes. In addition, potential applications of LSC in lifelong learning and NN distillation are discussed illustrating versatility of the proposed methodology.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。