用稀疏结构先验提升神经网络效率，让模型更省参数、更稳定。
Compositional Sparsity as an Inductive Bias for Neural Architecture Design
- 通过信息过滤与拓扑映射构建稀疏神经图，实现层级组合式结构设计
- 在高维任务中参数量减少多个数量级，且性能优于密集模型
- 适合追求高效、可解释模型的开发者，尤其适用于高维数据
识别使深度神经网络克服维度灾难的结构先验，是机器学习理论中的根本挑战。现有研究指出，有效高维学习依赖于组合稀疏性，即目标函数可分解为支持在低维变量子集上的组成部分。为此，我们结合信息过滤网络（IFNs）——通过受限信息最大化提取稀疏依赖结构——与同调神经网络（HNNs）——将推断出的拓扑映射为固定连接的稀疏神经图。我们形式化了该架构的设计原则，并提出一个可解释的流水线，其中抽象通过层级组合自然涌现。HNNs 的参数量远低于标准 DNN，且仅需极少超参数调优。在具有已知稀疏层级结构的合成任务中，HNNs 能准确恢复底层组合结构，在维度增加时仍保持稳定，而密集模型则性能急剧下降。在广泛的真实数据集上，HNNs 始终达到或超越密集基线，使用参数更少，方差更低，对超参数敏感度显著降低。
原文摘要 · Abstract (English)
Identifying the structural priors that enable Deep Neural Networks (DNNs) to overcome the curse of dimensionality is a fundamental challenge in machine learning theory. Existing literature suggests that effective high-dimensional learning is driven by compositional sparsity, where target functions decompose into constituents supported on low-dimensional variable subsets. To investigate this hypothesis, we combine Information Filtering Networks (IFNs), which extract sparse dependency structures via constrained information maximisation, with Homological Neural Networks (HNNs), which map the inferred topology into fixed-wiring sparse neural graphs. We formalise the design principles underlying this construction and present an interpretable pipeline in which abstraction emerges through hierarchical composition. HNNs are orders of magnitude sparser than standard DNNs and require only minimal hyperparameter tuning. On synthetic tasks with known sparse hierarchies, HNNs recover the underlying compositional structure and remain stable in regimes where dense alternatives degrade as dimensionality increases. Across a broad suite of real-world datasets, HNNs consistently match or outperform dense baselines while using far fewer parameters, exhibiting lower variance and showing reduced sensitivity to hyperparameters.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。