提出自适应降温策略，解决结构推断中梯度爆炸问题
Avoiding Premature Collapse: Adaptive Annealing for Entropy-Regularized Structural Inference
- 设计动态降温算法，实时监控推断过程稳定性
- 在FineWeb-Edu上训练时防止晚期梯度爆炸
- 适用于大规模结构化模型的稳定训练
可微分匹配层与残差连接常通过熵正则化最优传输（OT）实现，但在退火ε→0时恢复离散排列或保持恒等映射极不稳定。本文揭示其根本原因为‘过早模式坍缩’：通过分析Sinkhorn不动点映射的非正规动力学，发现理论热力学速度极限——标准指数降温快于推理算子收缩率，后者随1/ε衰减。为此，提出高效分段混合自适应稳定性控制（EPH-ASC），通过监测推断过程稳定性实现自适应调度。实验证明，EPH-ASC对在FineWeb-Edu数据集上的大规模训练中曼达帕约束超连接（mHC）至关重要，有效抑制晚期梯度爆炸，满足线性稳定性定律。
原文摘要 · Abstract (English)
Differentiable matching layers and residual connection paradigms, often implemented via entropy-regularized Optimal Transport (OT), serve as critical mechanisms in structural prediction and architectural scaling. However, recovering discrete permutations or maintaining identity mappings via annealing $ε\to 0$ is notoriously unstable. In this work, we identify a fundamental mechanism for this failure: \textbf{Premature Mode Collapse}. By analyzing the non-normal dynamics of the Sinkhorn fixed-point map, we reveal a theoretical thermodynamic speed limit: standard exponential cooling outpaces the contraction rate of the inference operator, which degrades as $O(1/ε)$. To address this, we propose \textbf{Efficient Piecewise Hybrid Adaptive Stability Control (EPH-ASC)}, an adaptive scheduling algorithm that monitors the stability of the inference process. We demonstrate that EPH-ASC is essential for stabilizing Manifold-Constrained Hyper-Connections (mHC) during large-scale training on the FineWeb-Edu dataset, effectively preventing late-stage gradient explosions by enforcing a linear stability law.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。