用分形和混沌激活函数提升神经网络在极端条件下的稳定性与速度
Beyond Lipschitz Continuity and Monotonicity: Fractal and Chaotic Activation Functions in Echo State Networks
- 引入分形、混沌等非光滑激活函数，突破传统平滑函数限制
- 柯尔特函数在谱半径达10时仍保持稳定，收敛速度比tanh快2.6倍
- 发现预处理拓扑比连续性更关键，适合高鲁棒性系统设计
当前回溯计算严重依赖平滑的全局Lipschitz连续激活函数，限制了其在国防、灾害应对和药物建模等极端条件下应用。我们系统研究了非光滑激活函数（包括混沌、随机和分形）在回声状态网络中的表现。通过36,610种网络配置的全面参数扫描，证明多个非光滑函数不仅维持回声状态特性（ESP），且在收敛速度和谱半径容限上优于传统平滑激活函数。值得注意的是，柯尔特函数（处处连续但几乎处处平坦）可在谱半径ρ≈10时保持一致的ESP行为，比平滑函数的典型上限高一个数量级，同时实现比tanh和ReLU快2.6倍的收敛速度。我们提出量化激活函数的理论框架，定义退化回声状态特性（d-ESP），证明d-ESP蕴含传统ESP。识别出关键拥挤比Q=N/k（网络规模/量化层级）可预测离散激活的失效阈值。分析表明，预处理拓扑结构而非连续性决定稳定性：单调压缩型预处理在各尺度维持ESP，而发散或不连续预处理导致突变失效。尽管研究挑战了激活函数设计的传统假设，某些分形函数表现出色的机制仍不明，揭示了对激活函数几何属性如何影响网络动态理解的根本缺失。
原文摘要 · Abstract (English)
Contemporary reservoir computing relies heavily on smooth, globally Lipschitz continuous activation functions, limiting applications in defense, disaster response, and pharmaceutical modeling where robust operation under extreme conditions is critical. We systematically investigate non-smooth activation functions, including chaotic, stochastic, and fractal variants, in echo state networks. Through comprehensive parameter sweeps across 36,610 reservoir configurations, we demonstrate that several non-smooth functions not only maintain the Echo State Property (ESP) but outperform traditional smooth activations in convergence speed and spectral radius tolerance. Notably, the Cantor function (continuous everywhere and flat almost everywhere) maintains ESP-consistent behavior up to spectral radii of rho ~ 10, an order of magnitude beyond typical bounds for smooth functions, while achieving 2.6x faster convergence than tanh and ReLU. We introduce a theoretical framework for quantized activation functions, defining a Degenerate Echo State Property (d-ESP) that captures stability for discrete-output functions and proving that d-ESP implies traditional ESP. We identify a critical crowding ratio Q=N/k (reservoir size / quantization levels) that predicts failure thresholds for discrete activations. Our analysis reveals that preprocessing topology, rather than continuity per se, determines stability: monotone, compressive preprocessing maintains ESP across scales, while dispersive or discontinuous preprocessing triggers sharp failures. While our findings challenge assumptions about activation function design in reservoir computing, the mechanism underlying the exceptional performance of certain fractal functions remains unexplained, suggesting fundamental gaps in our understanding of how geometric properties of activation functions influence reservoir dynamics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。