通过神经元竞争机制，让模型自动形成有功能专长的模块。
Sparse Competition during Training For the Emergence of Specialized Modules
- 训练时稀疏路由输入到神经元组，激发竞争
- 无需监督，模块激活与类别强相关，准确率接近基线
- 适合关注可解释性与模块化设计的研究者
深度神经网络中的模块化被认为能提升可解释性并改善训练，通过促进解耦表示和减少冗余。本文研究了训练过程中神经元组之间的竞争动态如何催生模块结构。提出一种方法：（i）保持接近基线的准确率；（ii）通过稀疏路由输入到神经元组，实现基于使用情况的模块化；（iii）促进模块专属性，使模块激活与输入类别相关。在ImageNet-100和CIFAR-100上验证，该方法可在无模块级监督下促使专用模块自发出现。这些模块捕捉数据中具有意义的高层结构，单个模块响应特定语义类别（如狗或车辆）。还研究了模块数量对子任务层次划分的影响。结果表明，竞争动态是诱导标准架构中功能性模块化的简单有效机制。
原文摘要 · Abstract (English)
Modularity in deep neural networks has been proposed as a means of improving both interpretability and training by promoting disentangled representations and reducing redundancy. In this work, we study the emergence of modular structure through competition dynamics between groups of neurons during training. We introduce a method that (i) maintains near-baseline accuracy, (ii) induces usage-based modularity by sparsely routing inputs to neuron groups, and (iii) encourages specialization of these modules, such that their activations are correlated with input classes. We evaluate the proposed approach on ImageNet-100 and CIFAR-100 and show that with it, specialized modules emerge without module-level supervision. These modules capture a meaningful high-level structure in the data, with individual modules responding to semantic categories (e.g., dogs or vehicles). We also study the emergence of a hierarchical partition of sub-tasks depending on the number of modules. Our results suggest that competitive dynamics can serve as a simple mechanism for inducing functional modularity in standard architectures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。