用随机梯度上升加速狄利克雷过程混合模型的在线学习,兼顾速度与性能。
Fisher Information based Stochastic Gradient Ascent for Online Learning of Dirichlet Process Mixture and Theory
- 基于费舍尔信息自适应步长,提升随机梯度上升的收敛效率
- 在Caltech256和SUN397数据集上保持与闭式坐标上升相当的性能
- 兼容深度卷积特征,适合大规模类别数据的在线学习
可扩展的后验近似算法使狄利克雷过程混合模型能够以分数成本处理更大规模数据集。尽管近期的随机变分推断通过小批量进行局部学习,但其依赖闭式解。本文探索使用随机梯度上升作为狄利克雷过程混合模型的快速后验近似方法。为兼顾速度与性能,我们引入步长优化:先采用动量法优化步长,最终结合费舍尔信息实现自适应步长。实验表明,该方法在保持与闭式坐标上升相当性能的同时,显著提升计算效率。此外,该方法可与深度卷积网络特征兼容,并适用于如Caltech256和SUN397等大规模类别数据集。
原文摘要 · Abstract (English)
Scalable algorithms of posterior approximation allow Bayesian nonparametrics such as Dirichlet process mixture to scale up to larger dataset at fractional cost. Recent algorithms, notably the stochastic variational inference performs local learning from minibatch. The main problem with stochastic variational inference is that it relies on closed form solution. Stochastic gradient ascent is a modern approach to machine learning and is widely deployed in the training of deep neural networks. In this work, we explore using stochastic gradient ascent as a fast algorithm for the posterior approximation of Dirichlet process mixture. However, stochastic gradient ascent alone is not optimal for learning. In order to achieve both speed and performance, we turn our focus to stepsize optimization in stochastic gradient ascent. As as intermediate approach, we first optimize stepsize using the momentum method. Finally, we introduce Fisher information to allow adaptive stepsize in our posterior approximation. In the experiments, we justify that our approach using stochastic gradient ascent do not sacrifice performance for speed when compared to closed form coordinate ascent learning on these datasets. Lastly, our approach is also compatible with deep ConvNet features as well as scalable to large class datasets such as Caltech256 and SUN397.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。