为Lion优化器设计分层学习率，提升ViT模型性能
LionVote: Per-Layer Learning Rate Adaptation for Lion
- 每层参数独立监控梯度与动量状态，动态调整学习率
- 在ViT-Tiny/CIFAR-100上达到69.7%准确率，优于Lion的69.0%
- 适用于结构差异大的模型，尤其在ViT架构中效果显著
分层诊断显示，在设定学习率下，Lion优化器对注意力层和MLP参数的有效尺度高出2.6-2.8倍，对归一化层高出约2倍，跨层类型差异达32%，无法用单一全局学习率模拟。该测量基于LionVote机制：每个参数张量维护一个复合等级，每c个周期通过梯度方向稳定性与动量健康度两项诊断更新，以验证损失为分歧解决依据。投票阈值由几何恒等式、指数移动平均时间常数及噪声下限估计决定；更新频率由结构约束并经消融实验选定。在ViT-Tiny/CIFAR-100上，LionVote实现69.7%的top-1准确率，优于Lion的69.0%（p < 0.02，Welch's t检验）和AdamW的68.8%。分层适应价值取决于架构异质性和任务特性；在均一CNN架构中，使用余弦退火的SGD仍占优，而ViT架构中的收益具有任务依赖性。
原文摘要 · Abstract (English)
Per-layer diagnostics reveal that, at the prescribed learning rate, Lion's effective scale is 2.6-2.8x too high for attention and MLP parameters and ~2x too high for normalization layers on ViT-Tiny/CIFAR-100; this 32% cross-layer-type disparity cannot be reproduced by a single global rate. The measurement comes from LionVote, a per-layer learning rate mechanism in which each parameter tensor maintains a compound level, a persistent integer updated every c epochs by two diagnostics (gradient direction stability and momentum health) resolved by a validation loss tiebreaker. Voting thresholds derive from geometric identities, the EMA time constant, and a noise-floor estimate; cadence is bounded structurally and selected by ablation. On ViT-Tiny/CIFAR-100, LionVote achieves 69.7% top-1 accuracy vs. Lion's 69.0% (p < 0.02, Welch's t-test) and AdamW's 68.8%. Per-layer adaptation value depends on both architectural heterogeneity and task; on uniform CNN architectures tuned SGD with cosine annealing remains dominant, and on ViT architectures gains are task-dependent.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。