让无法自动求导的模块也能参与端到端训练
BOND: License to Train with Black-Box Functions
- 通过自适应限制扰动范围，精准估计黑箱函数梯度
- 在冻结网络模块下提升模型性能，不增加参数量
- 适合想用固定变换扩展模型能力的研究者
我们提出有界数值微分（BOND），一种用于估计黑箱函数梯度的扰动方法。BOND 的设计能自适应地约束扰动大小，确保梯度符号的准确估计，且在黑箱接口上实现，具备更高精度与可扩展性。相比现有方法，它支持包含非自动微分模块的架构进行端到端训练。实验表明，将这些模块以冻结网络形式实现，可在不增加可训练参数的前提下提升模型表现。结果揭示了利用固定变换扩展模型容量的潜力，为混合模拟-数字设备拓展网络规模提供新路径，并深化了对自适应优化器动态机制的理解。
原文摘要 · Abstract (English)
We introduce Bounded Numerical Differentiation (BOND), a perturbative method for estimating the gradients of black-box functions. BOND is distinguished by its formulation, which adaptively bounds perturbations to ensure accurate sign estimation, and by its implementation, which operates at black-box interfaces. This enables BOND to be more accurate and scalable compared to existing methods, facilitating end-to-end training of architectures that incorporate non-autodifferentiable modules. We observe that these modules, implemented in our experiments as frozen networks, can enhance model performance without increasing the number of trainable parameters. Our findings highlight the potential of leveraging fixed transformations to expand model capacity, pointing to hybrid analogue - digital devices as a path to scaling networks, and provides insights into the dynamics of adaptive optimizers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。