无需提前知道参数,也能达到最优优化效率。
The Sample Complexity of Parameter-Free Stochastic Convex Optimization
- 设计可靠模型选择方法,避免小验证集过拟合。
- 通过正则化估计最优解距离,实现样本复杂度最优。
- 适用于参数未知场景,尤其适合少样本学习任务。
研究当问题参数(如最优解距离、Lipschitz常数)未知时,随机凸优化的样本复杂度。提出两种策略:其一,开发可靠的模型选择方法,避免对小验证集过拟合,可使随机优化方法的学习率自适应调整至已知参数下的最优样本复杂度,仅相差 log log 因子;其二,针对仅最优解距离未知的情况,采用范数正则化的经验风险最小化,将最优解距离估计到常数因子内,从而使已有优化方法达到最优样本复杂度。该方法展示了参数无关优化中样本复杂度与计算复杂度的分离。结合两者可同时适应多个未知结构。在CIFAR-10上进行少样本微调和Gemini提示计数形状的实验表明,所提方法能有效缓解小验证集导致的过拟合。
原文摘要 · Abstract (English)
We study the sample complexity of stochastic convex optimization when problem parameters such as the distance to optimality and the Lipschitz constant are unknown. We pursue two strategies. First, we develop a reliable model selection method that avoids overfitting to the validation set. This method allows us to generically tune the learning rate of stochastic optimization methods to match the optimal known-parameter sample complexity up to log log factors. Second, we develop a regularization-based method that is specialized to the case that only the distance to optimality is unknown. More specifically, it uses norm-regularized empirical risk minimization to estimate the distance to optimality to within a constant factor, allowing known-parameter stochastic optimization methods to achieve optimal sample complexity. This method provides perfect adaptability to unknown distance to optimality, demonstrating a separation between the sample and computational complexity of parameter-free stochastic convex optimization. Combining these two methods allows us to simultaneously adapt to multiple problem structures. Experiments performing few-shot learning on CIFAR-10 by fine-tuning CLIP models and prompt engineering Gemini to count shapes indicate that our reliable model selection method can help mitigate overfitting to small validation sets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。