用短时试跑动态选最优优化器，省时且效果接近穷举搜索。
Many Optimizers But Only One Training Path: Repeated Resampling for Adaptive Optimizer Selection
- 每 b 个周期内，多个优化器并行试跑 s 个周期，表现最好者继续
- 一周期试跑仅需固定优化器总训练量的24%~35%，性能接近最优
- 适合想省训练时间又不想手动调优的人
优化器通常在训练前选定并保持不变。将优化器选择视为超参数可提升性能，但需多次完整训练且只保留最优结果。重复优化器重采样（ROR）则在单个持续训练中完成搜索：每 b 个周期，各候选优化器从当前模型权重出发，进行 s 个周期试跑；表现最佳者继续后续 b−s 个周期，并在验证目标改善时成为新基准。该设计允许优选优化器随训练过程变化。我们在 MNIST、Fashion-MNIST 及两个保险索赔计数模型上对比了 ROR 的两种变体，评估了九种固定优化器与两种 ROR 变体，使用相同十组随机种子。一周期 ROR 仅需固定优化器总训练量的 24% 到 35%，即可在四个任务上接近穷举搜索得出的最佳固定优化器，表明短时试跑是无需完成全部训练即可高效搜索优化器的有效方法。
原文摘要 · Abstract (English)
An optimizer is usually chosen before training a deep neural network and then kept fixed. Treating optimizer choice as a hyperparameter could boost performance, but it requires several complete training runs and discards all but the winner. Repeated Optimizer Resampling (ROR) instead searches during one evolving run. Every $b$ epochs, each candidate optimizer scouts from the current model weights for $s$ epochs. The best scout continues for the remaining $b-s$ epochs, and that completed segment becomes the new incumbent if it improves the validation objective. This design allows the preferred optimizer to change as training progresses. We compare two variants of ROR on MNIST, Fashion-MNIST, and two motor insurance claim-count models. Nine fixed optimizers and both ROR variants are evaluated with the same ten seeds. One-epoch ROR uses 24\% to 35\% of the aggregate training needed to identify the best fixed optimizer exhaustively and remains close to that optimizer on all four tasks. These results support short scouting as a practical way to search over optimizers without completing every candidate run.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。