arXiv:2601.13160cs.LGcs.AI2026-01被引 1

发现深度学习训练不稳有低维动力学规律,可预测崩溃前兆。

Training instability in deep learning follows low-dimensional dynamical principles

  • 从优化、数据、参数、学习信号四维度构建训练稳定性统一视角
  • 高最终性能常与训练不稳定解耦,随机性可缓冲动态波动
  • 低维潜在状态偏移是性能崩溃的早期预警信号

深度学习系统虽表现卓越,但训练过程的稳定性仍不清楚。训练本质是高维动力系统,微小扰动(如优化、数据、参数或学习信号)可能引发突然且不可逆的崩溃,影响可复现性和可扩展性。本文提出统一的动力学视角,将训练稳定性视为学习系统的内在属性,涵盖优化、环境/数据、参数和学习信号四个相互作用维度。通过受控扰动审计训练轨迹,探究学习动态对结构化干扰的响应,不修改学习算法。在强化学习与大语言模型训练中,识别出三个普遍规律:高最终性能常与训练不稳定解耦;可控随机性在各范式中持续缓冲学习动态;低维潜在元状态的偏离系统性地先于可观测性能崩溃出现。这些发现确立了训练稳定性作为可测量、可比较的学习系统动力学属性,为超越最终性能结果的研究提供描述性基础。

原文摘要 · Abstract (English)

Deep learning systems achieve remarkable empirical performance, yet the stability of the training process itself remains poorly understood. Training unfolds as a high-dimensional dynamical system in which small perturbations to optimization, data, parameters, or learning signals can induce abrupt and irreversible collapse, undermining reproducibility and scalability. We propose a unified dynamical perspective that characterizes training stability as an intrinsic property of learning systems, organized along four interacting dimensions: optimization, environmental/data, parametric, and learning-signal stability. We operationalize this perspective through controlled perturbation auditing of training trajectories, probing how learning dynamics respond to structured disturbances without modifying learning algorithms. Across reinforcement learning and large language model training, we identify three recurring regularities: high final performance is frequently decoupled from training stability; controlled stochasticity consistently buffers learning dynamics across paradigms; and deviations in low-dimensional latent meta-states systematically precede observable performance collapse. Together, these findings establish training stability as a measurable and comparable dynamical property of learning systems, providing a descriptive foundation for studying learning dynamics beyond final performance outcomes.

训练稳定动力系统大模型崩溃预警

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。