提出分层零阶优化，显著降低深度网络的查询成本。
Hierarchical Zero-Order Optimization for Deep Neural Networks
- 将网络深度分解为层级优化，替代逐层传播梯度
- 查询复杂度从O(ML²)降至O(ML log L)，提升效率
- 适合非可导目标或生物启发场景，对深度网络有效
零阶（ZO）优化因具备生物合理性及处理不可导目标的能力而备受青睐，但其计算开销长期限制其在深度神经网络中的应用。我们挑战了传统梯度逐层传播范式，提出分层零阶（HZO）优化，一种将网络深度维度分解的新型分治策略。理论证明，对于宽度为M、深度为L的网络，HZO将查询复杂度从O(ML²)降低至O(ML log L)，显著优于现有零阶方法。此外，详细误差分析表明，HZO通过在单位限附近运行（L_{lip} ≈ 1），保持数值稳定性。在CIFAR-10和ImageNet上的大量实验表明，HZO在精度上可与反向传播相媲美。
原文摘要 · Abstract (English)
Zeroth-order (ZO) optimization has long been favored for its biological plausibility and its capacity to handle non-differentiable objectives, yet its computational complexity has historically limited its application in deep neural networks. Challenging the conventional paradigm that gradients propagate layer-by-layer, we propose Hierarchical Zeroth-Order (HZO) optimization, a novel divide-and-conquer strategy that decomposes the depth dimension of the network. We prove that HZO reduces the query complexity from $O(ML^2)$ to $O(ML \log L)$ for a network of width $M$ and depth $L$, representing a significant leap over existing ZO methodologies. Furthermore, we provide a detailed error analysis showing that HZO maintains numerical stability by operating near the unitary limit ($L_{lip} \approx 1$). Extensive evaluations on CIFAR-10 and ImageNet demonstrate that HZO achieves competitive accuracy compared to backpropagation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。