arXiv:2606.19941cs.LG2026-06

模型的组合性只在特定深度和连接度下出现,突破传统认知。

Compositionality Emerges in a Narrow Depth-Connectivity Regime: Architecture Constraints and Solution Manifolds

论文配图:Compositionality Emerges in a Narrow Depth-Connectivity Regime: Architecture Constraints and Solution Manifolds
图 1 · 摘自论文原文
  • 发现组合性仅在特定稀疏连接与适中深度时涌现
  • 过浅或过深、连接过密都会导致结构断裂
  • 提出剪枝与深度预测法,可主动引导组合结构形成

组合性被认为是泛化能力的基础,使模型能复用有意义的组件进行新组合。然而,标准梯度优化训练的模型很少甚至几乎不表现出强组合性,其形成机制尚不清楚。本文表明,组合性仅在狭窄的连接-深度区间内出现:连接上需特定稀疏结构(非单纯权重稀疏),深度上存在目标依赖的峰值区域,过浅或过深均失效。任一条件被破坏,梯度下降将收敛至非组合的碎片解。为此,我们提出基于相似性的剪枝(SP)以恢复组合连接,并设计启发式深度预测器估计组合性最可能出现的位置。最后,通过组合稀疏性、体积比论证和特征干扰界,建立理论框架,解释为何组合解仅在狭小深度-连接区间可达。

原文摘要 · Abstract (English)

Compositionality is believed to be the foundation for generalization, enabling models to reuse meaningful primitives in novel combinations. Yet, models trained with standard gradient-based optimization rarely, and often only weakly, exhibit compositional internal structure, and it remains unclear how or why such compositionality forms. In this work, we show that compositionality emerges in a narrow connectivity-depth sweet spot. Along the connectivity axis, compositionality only appears in some specifically sparse networks, heavily depends on which connections remain rather than on weights' sparsity alone. Along the depth axis, compositionality emerges within a narrow, target-dependent regime, peaking at specific depths, while both shallower and deeper networks fail. When either the depth or connectivity condition is violated, gradient descent silently converges to fractured solutions rather than compositional ones. To discover and exploit this emergence, we introduce (i) similarity-based pruning (SP) to recover compositional connectivity and (ii) a heuristic depth predictor to estimate where compositionality is most likely to appear. Finally, we support these empirical findings with a theoretical framework based on compositional sparsity, volume-ratio arguments, and feature-interference bounds, explaining why compositional solutions are reachable only in a narrow depth-connectivity regime.

组合性神经网络架构深度学习机制结构涌现

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。