用信息论新指标解释优化器与网络结构的适配机制
Information-Theoretic Perspectives on Optimizers
- 引入熵差指标替代传统尖锐度衡量优化器性能
- 发现尖锐度与熵差共同影响优化动态和泛化能力
- 基于信息论改进Lion优化器，提升训练效果
神经网络中优化器与架构的相互作用复杂且难以理解，为何某些优化器在特定架构上表现更优仍不明确。本文指出，传统尖锐度指标无法充分解释这一复杂关系，提出信息论新度量——熵差（entropy gap），以更深入分析该问题。研究发现，尖锐度与熵差均影响模型的优化动态和泛化性能。进一步利用信息论工具分析近期提出的Lion优化器，揭示其内在机制，并提出改进方向，显著提升其训练稳定性与收敛性。
原文摘要 · Abstract (English)
The interplay of optimizers and architectures in neural networks is complicated and hard to understand why some optimizers work better on some specific architectures. In this paper, we find that the traditionally used sharpness metric does not fully explain the intricate interplay and introduces information-theoretic metrics called entropy gap to better help analyze. It is found that both sharpness and entropy gap affect the performance, including the optimization dynamic and generalization. We further use information-theoretic tools to understand a recently proposed optimizer called Lion and find ways to improve it.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。