INNAprop融合二阶信息与自适应梯度缩放，训练更快更准。
A second-order-like optimizer with adaptive gradient scaling for deep learning
- 结合INNA的二阶信息与RMSprop的自适应缩放机制。
- 在多个数据集和模型上超越AdamW，且调参极少。
- 适合追求高效训练的深度学习研究者与工程师。
本文提出INNAprop优化算法，融合INNA方法与RMSprop的自适应梯度缩放，利用二阶信息并保持与AdamW或SGD with momentum相当的内存开销。通过几何分析，我们在CIFAR-10、Food101、ImageNet上使用ResNet、VGG、DenseNet和ViT进行评估，并在GPT-2（OpenWebText）上从头训练及使用LoRA进行端到端微调。实验表明，INNAprop在大规模设置下训练速度与准确率均匹配或优于AdamW，且超参数调整极小。代码已公开于https://github.com/innaprop/innaprop。
原文摘要 · Abstract (English)
In this empirical article, we introduce INNAprop, an optimization algorithm that combines the INNA method with the RMSprop adaptive gradient scaling. It leverages second-order information and rescaling while keeping the memory requirements of standard DL methods as AdamW or SGD with momentum. After giving geometrical insights, we evaluate INNAprop on CIFAR-10, Food101, and ImageNet with ResNets, VGG, DenseNet, and ViT, and on GPT-2 (OpenWebText) train from scratch and with LoRA fine-tuning (E2E). INNAprop consistently matches or outperforms AdamW both in training speed and accuracy, with minimal hyperparameter tuning in large-scale settings. Our code is publicly available at \url{https://github.com/innaprop/innaprop}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。