神经网络先学边缘分布，再突变学习条件分布。
Marginals Before Conditionals
- 构建最小任务，分离条件学习机制。
- 边缘分布熵为 log K，出现稳定平台期。
- 梯度噪声稳定边缘解，触发条件学习需克服熵力。
我们设计了一个最小化任务，以隔离神经网络中的条件学习过程：一个具有K重歧义的满射映射，由选择标记z解决，使得H(A|B) = log K，而H(A|B,z) = 0。模型首先学习边缘分布P(A|B)，在精确log K处形成平台，随后经历一次急剧的集体跃迁完成完整条件学习。该平台具有清晰分解特性：高度等于log K（由歧义决定），持续时间取决于数据集大小D，而非K。梯度噪声稳定边缘解：更高学习率单调减缓跃迁速度（在固定吞吐量下η变化7倍时，速度下降3.6倍），批量减小延迟脱离平台，符合熵力对抗低梯度边缘解的理论。内部观察发现，选择路由头在平台期逐步形成，其损失下降领先于总损失约50%的等待时间。这对应于Papadopoulos等人[2024]提出的类型2方向不对称性，我们动态追踪从log K到零的超额风险，并刻画其稳定性、崩溃触发机制及持续时间。
原文摘要 · Abstract (English)
We construct a minimal task that isolates conditional learning in neural networks: a surjective map with K-fold ambiguity, resolved by a selector token z, so H(A | B) = log K while H(A | B, z) = 0. The model learns the marginal P(A | B) first, producing a plateau at exactly log K, before acquiring the full conditional in a sharp, collective transition. The plateau has a clean decomposition: height = log K (set by ambiguity), duration = f(D) (set by dataset size D, not K). Gradient noise stabilizes the marginal solution: higher learning rates monotonically slow the transition (3.6* across a 7* η range at fixed throughput), and batch-size reduction delays escape, consistent with an entropic force opposing departure from the low-gradient marginal. Internally, a selector-routing head assembles during the plateau, leading the loss transition by ~50% of the waiting time. This is the Type 2 directional asymmetry of Papadopoulos et al. [2024], measured dynamically: we track the excess risk from log K to zero and characterize what stabilizes it, what triggers its collapse, and how long it takes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。