探索分数阶优化与分形激活函数的协同效应,发现特定组合可提升训练性能。
Fractional Optimizers Meet Fractal Activation Functions: An Empirical Study of Multi-Scale Optimization in Neural Network

- 将分数阶优化与分形激活结合,在多尺度上增强神经网络表示能力。
- 在扰动优化表面和10个分类数据集上,部分组合显著提升训练效果。
- 自适应记忆机制优于简单替换,适合特定任务而非通用方案。
分数阶优化方法与分形激活函数是提升神经网络训练性能的两个独立方向。前者通过分数阶导数和记忆效应扩展一阶优化,后者基于自相似的Weierstrass型和Blancmange型函数引入多尺度非线性表征。本文在统一实验框架下研究二者交互作用:评估分数阶优化器在Ackley与Himmelblau基准曲面(含Weierstrass型扰动)及十类分类数据集上的表现,对比标准方法、正则化风格优化器、显式与自适应记忆型分数阶优化器等。结果表明,分数阶优化与分形激活存在选择性协同效应:正则化风格的分数阶缩放在部分分形激活下表现优异,Grünwald-Letnikov记忆在扰动表面上更有效;自适应记忆在多个场景中优于普通记忆替换,支持受控分数阶记忆作为有前景的方向,而非通用替代方案。
原文摘要 · Abstract (English)
Fractional optimization methods and fractal activation functions are two independent directions for improving neural network training. Fractional optimizers extend first-order optimization through fractional derivatives and memory effects, whereas fractal activations introduce multi-scale nonlinear representations based on self-similar Weierstrass- and Blancmange-type functions. Here, we investigate their interaction within a unified experimental framework. We evaluate fractional optimizer families on Ackley and Himmelblau benchmark surfaces, in standard form and with additive Weierstrass-type perturbations, and then in feed-forward neural networks with conventional and fractal activations on ten classification datasets. The comparison includes standard methods, regularization-style optimizers, explicit and adaptive memory-based fractional optimizers, and other representative literature methods. Overall, fractional optimization and fractal activations show useful but selective pairings. Regularization-style fractional scaling performs well with selected fractal activations in network training, while Grünwald--Letnikov memory is most relevant on perturbed surfaces. Adaptive memory improves plain memory substitution in several cases, supporting controlled fractional memory as a promising direction rather than a universal replacement.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。