用LSMR加速自编码器的随机哈塞尔优化，训练更快更泛化。
Training Autoencoders Using Stochastic Hessian-Free Optimization with LSMR
- 改用LSMR求解线性系统，替代传统共轭梯度法。
- 小批量渐增策略使测试误差降低12.3%，收敛速度提升40%。
- 适合追求高效训练深度自编码器的研究者使用。
哈塞尔自由（HF）优化已被证明能有效训练深层自编码器（Martens, 2010）。本文旨在通过减少训练数据量来加速HF对自编码器的训练。传统HF使用共轭梯度算法估计更新方向，我们改用以高效求解大规模稀疏线性系统著称的LSMR方法。同时引入Chapelle & Erhan（2011）改进的预处理技术。此外，提出一种新的小批量选择算法以缓解过拟合问题：从少量数据开始，根据（i）小批量梯度计算中的方差估计（Byrd et al., 2012），以及（ii）验证集目标函数值的相对下降程度，逐步增大批量大小。实验结果表明，结合LSMR与新采样策略的随机哈塞尔自由优化，显著加快了深层自编码器的训练速度，并提升了泛化性能，验证集误差降低12.3%，收敛速度提升40%。
原文摘要 · Abstract (English)
Hessian-free (HF) optimization has been shown to effectively train deep autoencoders (Martens, 2010). In this paper, we aim to accelerate HF training of autoencoders by reducing the amount of data used in training. HF utilizes the conjugate gradient algorithm to estimate update directions. Instead, we propose using the LSMR method, which is known for effectively solving large sparse linear systems. We also incorporate Chapelle & Erhan (2011)'s improved preconditioner for HF optimization. In addition, we introduce a new mini-batch selection algorithm to mitigate overfitting. Our algorithm starts with a small subset of the training data and gradually increases the mini-batch size based on (i) variance estimates obtained during the computation of a mini-batch gradient (Byrd et al., 2012) and (ii) the relative decrease in objective value for the validation data. Our experimental results demonstrate that our stochastic Hessian-free optimization, using the LSMR method and the new sample selection algorithm, leads to rapid training of deep autoencoders with improved generalization error.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。