用时间动态约束提升模型泛化能力,类比生物系统机制
Constraint Breeds Generalization: Temporal Dynamics as an Inductive Bias
- 引入时间动态约束作为归纳偏置,替代传统无约束优化
- 在分类、重建和零样本强化学习中均发现泛化能力的临界跃迁
- 适合追求鲁棒性与泛化的深度学习研究者,尤其关注时序建模
传统深度学习强调无约束优化,但生物系统受严格代谢约束。我们提出,这些物理约束塑造动态过程,非为限制,而是形成促进泛化的时序归纳偏置。通过信号传播的相空间分析,揭示基本不对称性:发散动态放大噪声,而恰当耗散动态压缩相空间,契合网络谱偏置,促使不变特征抽象。该条件可通过输入编码外部施加,或通过网络自身时序动态内在实现。两条路径均需具备时序整合能力及恰当约束以解码诱导的不变性,静态架构则无法利用时序结构。在监督分类、无监督重构和零样本强化学习中综合评估,均显示存在一个最大化泛化能力的‘临界’状态。研究确立动态约束为一类独立归纳偏置,表明构建鲁棒人工智能不仅需规模扩展与去除限制,更需计算上驾驭天然促进泛化的时序特性。
原文摘要 · Abstract (English)
Conventional deep learning prioritizes unconstrained optimization, yet biological systems operate under strict metabolic constraints. We propose that these physical constraints shape dynamics to function not as limitations, but as a temporal inductive bias that breeds generalization. Through a phase-space analysis of signal propagation, we reveal a fundamental asymmetry: expansive dynamics amplify noise, whereas proper dissipative dynamics compress phase space that aligns with the network's spectral bias, compelling the abstraction of invariant features. This condition can be imposed externally via input encoding, or intrinsically through the network's own temporal dynamics. Both pathways require architectures capable of temporal integration and proper constraints to decode induced invariants, whereas static architectures fail to capitalize on temporal structure. Through comprehensive evaluations across supervised classification, unsupervised reconstruction, and zero-shot reinforcement learning, we demonstrate that a critical "transition" regime maximizes generalization capability. These findings establish dynamical constraints as a distinct class of inductive bias, suggesting that robust AI development requires not only scaling and removing limitations, but computationally mastering the temporal characteristics that naturally promote generalization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。