arXiv:2503.06917cs.LGcs.DS2025-03被引 2

用粗略可学习性实现高效生成先验优化,样本数仅需多项式级。

Sample-Efficient Optimization over Generative Priors via Coarse Learnability

  • 引入'粗略可学习性'假设,放宽对生成模型精度要求。
  • 设计迭代算法 extsc{Lift},在多项式样本下逼近目标分布。
  • 适用于非凸优化,适合需要低样本代价的生成模型调优场景。

研究零阶优化问题:在保持复杂生成先验 $L(s)$ 高概率的前提下最小化成本 $d(s)$,等价于从与 $L(s) e^{-T ullet d(s)}$ 成比例的目标分布中采样。由于经典模型基于优化(MBO)缺乏对高表达力近似学习器的有限样本保证,本文提出‘粗略可学习性’这一灵活统计假设,仅要求学习模型覆盖目标概率质量至多项式因子内。基于此,设计了带样本修正步骤的迭代MBO算法 extsc{Lift},可证明以多项式数量样本逼近目标分布。将该框架应用于 $\mathbb{R}^n$ 中受二次包络约束的非凸目标全局优化,证明此类‘乐观后验分布’族自然满足该假设。达到全局 $\varepsilon$-最优时,样本复杂度为 $\widetilde{O}(\log 1/\varepsilon)$,符合乐观空间划分方法特征。进一步从理论上验证粗略可学习性在简单设定下成立,参数最大似然估计与过平滑核密度估计均满足该假设。最后,动机之一来自推理时对齐;虽主要贡献为MBO理论基础,但在简单设置中提供定性证据:即使原始LLM也能通过零阶反馈微调,使分布向低代价区域迁移。

原文摘要 · Abstract (English)

We study zeroth-order optimization where solutions must minimize a cost $d(s)$ while maintaining high probability under a complex generative prior $L(s)$ (e.g., a parameterized model). This reduces to sampling from a target distribution proportional to $L(s) e^{-T \cdot d(s)}$. Since classical model-based optimization (MBO) lacks finite-sample guarantees for expressive approximate learners, we introduce "coarse learnability", a flexible statistical assumption requiring only that a learned model covers the target's probability mass within a polynomial factor. Leveraging this assumption, we design an iterative MBO algorithm called \alift with a sample correction step that provably approximates the target using only a polynomial number of samples. We apply this framework to globally optimizing non-convex objectives bounded by a quadratic envelope in $R^n$, where we show this assumption is naturally satisfied for a family of "optimistic" posterior distributions. To reach global $\varepsilon$-optimality, this implies a sample complexity of $\widetilde{O}(\log 1/\varepsilon)$, a rate characteristic of optimistic space-partitioning methods. We further justify coarse learnability as an assumption for generative priors theoretically, proving that in simple settings, parametric maximum likelihood estimation and over-smoothed kernel density estimators naturally satisfy it. Finally, one motivation for our framework comes from inference-time alignment. Though our primary contribution pertains to the theoretical foundations of MBO, we provide qualitative evidence that, in simple settings, even primitive LLMs can shift their distributions toward lower-cost regions when fine-tuned with zeroth-order feedback.

零阶优化生成模型样本效率理论分析

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。