数据复杂度影响神经网络解附近低损失区域的分布形态。
Dataset Complexity Shapes Finite-Distance Loss Geometry in Neural Networks
- 用局部熵衡量解附近低损失参数空间的有效体积。
- 数据越复杂，解附近局部熵下降越显著，收缩位置更集中。
- 适用于研究模型泛化与损失曲面结构的学者。
有限数据集可能具有相同的大小和低阶统计特征，但结构复杂度差异显著。本文通过将局部标签混合与训练后神经网络解附近的局部熵结合，建立了数据复杂度与损失曲面几何的关系。借鉴自自旋玻璃理论的Franz--Parisi构造，局部熵用于度量距参考点不同距离处低损失、解类参数配置的有效体积。在有限网络中，采用自适应序列蒙特卡洛进行估计。在可控的合成实验中，数据复杂度越高，参考点附近的局部熵下降越明显；而远离参考点时，其径向导数趋于平缓且各条件间趋同。因此，数据复杂度改变的是有效解空间收缩的位置，而非均匀加速收缩。真实图像数据上的实验也显示相同定性趋势，标签随机化进一步放大该效应。结果表明，数据结构决定了低损失邻域在训练解周围有限距离上的组织方式。
原文摘要 · Abstract (English)
Finite datasets can share the same size and low-order statistics while differing strongly in structural complexity. We connect this dataset complexity to loss-landscape geometry by pairing local label mixing across neighborhood scales with local entropy around trained neural-network solutions. Adapted from the Franz--Parisi construction in spin-glass theory, local entropy measures the effective volume of low-loss, solution-like parameter configurations at each distance from a reference. We estimate it in finite networks using adaptive sequential Monte Carlo. In a controlled synthetic sweep, greater dataset complexity produces a larger decrease in local entropy near the reference. Farther away, its radial derivative becomes weak and nearly common across conditions. Dataset complexity therefore changes where the effective solution volume contracts, rather than making it decrease uniformly faster. Experiments on real image data show the same qualitative trend, with label randomization further amplifying the effect. These results show that dataset structure shapes how low-loss neighborhoods are organized across finite distances from trained solutions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。