提出高效随机数生成方法，让设备端零阶优化更可行
Perturbation-efficient Zeroth-order Optimization for Hardware-friendly On-device Training
- 复用随机数减少生成需求，改用均匀分布替代高斯分布
- 硬件资源消耗降48.6%（LUTs）和12.7%（FFs），功耗最高省86%
- 首次实现设备端零阶训练可行性，适合边缘计算研究者
零阶（ZO）优化是一种新兴的深度神经网络训练范式，具有计算简单和内存节省的优点。然而，这一看似有前景的方法面临一个长期被忽视的关键挑战：ZO需要生成大量高斯随机数，这对硬件平台（如FPGA和ASIC）造成巨大困难，甚至使其不可行。本文识别出算法与硬件设计者之间的不匹配问题，提出PeZO——一种扰动高效的零阶优化框架。具体而言，我们设计了随机数复用策略，显著降低随机数生成需求，并引入硬件友好的自适应缩放方法，以均匀分布替代代价高昂的高斯分布。实验表明，PeZO将随机数生成所需的LUTs和FFs分别减少48.6%和12.7%，最大可节省86%的功耗，且不影响训练性能，使零阶优化在设备端训练中成为可能。据我们所知，这是首个探索设备端零阶优化潜力的工作，为未来研究提供了重要启示。
原文摘要 · Abstract (English)
Zeroth-order (ZO) optimization is an emerging deep neural network (DNN) training paradigm that offers computational simplicity and memory savings. However, this seemingly promising approach faces a significant and long-ignored challenge. ZO requires generating a substantial number of Gaussian random numbers, which poses significant difficulties and even makes it infeasible for hardware platforms, such as FPGAs and ASICs. In this paper, we identify this critical issue, which arises from the mismatch between algorithm and hardware designers. To address this issue, we proposed PeZO, a perturbation-efficient ZO framework. Specifically, we design random number reuse strategies to significantly reduce the demand for random number generation and introduce a hardware-friendly adaptive scaling method to replace the costly Gaussian distribution with a uniform distribution. Our experiments show that PeZO reduces the required LUTs and FFs for random number generation by 48.6\% and 12.7\%, and saves at maximum 86\% power consumption, all without compromising training performance, making ZO optimization feasible for on-device training. To the best of our knowledge, we are the first to explore the potential of on-device ZO optimization, providing valuable insights for future research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。