通过周期性低秩海森子空间预条件，加速优化器在尖锐区域的收敛速度。
CAO: Curvature-Adaptive Optimization via Periodic Low-Rank Hessian Sketching
- 周期性用海森向量积构造低秩海森子空间，仅在此子空间预条件梯度
- 在CIFAR-100/ResNet-18上比Adam快2.95倍达到指定损失阈值
- 仅需调一个参数（子空间秩），对秩不敏感且可退化为无曲率方法
一阶优化器在陡峭、各向异性区域表现可靠但收敛慢。本文提出一种曲率自适应方法：通过海森-向量积周期性地构建低秩海森子空间，仅在该子空间内对梯度进行预条件，其余部分仍保持一阶。对于L-光滑非凸目标函数，恢复标准O(1/T)平稳性保证，并扩展了稳定步长范围；在存在有界残差曲率的Polyak–Lojasiewicz（PL）条件下，损失在刷新步骤处收缩。在CIFAR-10/100上使用ResNet-18/34实验显示，该方法显著更早进入低损失区域：以达到预设训练损失阈值0.75所需训练轮数衡量，在CIFAR-100/ResNet-18上比Adam快2.95倍，同时保持最终测试精度。该方法仅需调节一个参数（子空间秩k），在k∈{1,3,5}时性能稳定，且当k=0时可退化为合理的无曲率对照实验。论文附带匿名日志与脚本，可复现所有图表。
原文摘要 · Abstract (English)
First-order optimizers are reliable but slow in sharp, anisotropic regions. We study a curvature-adaptive method that periodically sketches a low-rank Hessian subspace via Hessian--vector products and preconditions gradients only in that subspace, leaving the orthogonal complement first-order. For L-smooth non-convex objectives, we recover the standard O(1/T) stationarity guarantee with a widened stable stepsize range; under a Polyak--Lojasiewicz (PL) condition with bounded residual curvature outside the sketch, the loss contracts at refresh steps. On CIFAR-10/100 with ResNet-18/34, the method enters the low-loss region substantially earlier: measured by epochs to a pre-declared train-loss threshold (0.75), it reaches the threshold 2.95x faster than Adam on CIFAR-100/ResNet-18, while matching final test accuracy. The approach is one-knob: performance is insensitive to the sketch rank k across {1,3,5}, and k=0 yields a principled curvature-free ablation. We release anonymized logs and scripts that regenerate all figures and tables.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。