arXiv:2602.16796cs.LGmath.OC2026-02被引 5

让生成模型同时控制极端低效和高效样本,提升可靠性与创新性。

Efficient Tail-Aware Generative Optimization via Flow Model Fine-Tuning

  • 基于条件风险价值(CVaR)设计分阶段微调流程,分离阈值优化与奖励学习。
  • 在文本到图像生成和分子设计中,显著提升高回报罕见样本的生成率。
  • 计算开销接近标准方法,适合实际部署场景,尤其适用于高风险领域。

将预训练扩散模型和流模型微调以优化下游任务性能,是实际应用的核心。现有基于熵正则的方法主要最大化期望奖励,缺乏对尾部行为的调控能力。然而,尾部控制至关重要:下尾决定可靠性,限制低回报失败;上尾促进发现,优先生成稀有高回报结果。本文提出一种基于条件风险价值(CVaR)的尾部感知流模型微调方法(TFFT),针对右尾(高奖励)探索和左尾(低奖励)控制分别设计优化目标。不同于依赖非线性优化的先前方法,我们利用CVaR的变分对偶形式,将其分解为轻量级一维阈值优化与单次熵正则微调,通过特定伪奖励实现高效解耦。该方法在文本到图像生成和分子设计等高维任务中验证有效,计算成本与标准期望微调相当。

原文摘要 · Abstract (English)

Fine-tuning pre-trained diffusion and flow models to optimize downstream utilities is central to real-world deployment. Existing entropy-regularized methods primarily maximize expected reward, providing no mechanism to shape tail behavior. However, tail control is often essential: the lower tail determines reliability by limiting low-reward failures, while the upper tail enables discovery by prioritizing rare, high-reward outcomes. In this work, we present Tail-aware Flow Fine-Tuning (TFFT), a principled and efficient distributional fine-tuning algorithm based on the Conditional Value-at-Risk (CVaR). We address two distinct tail-shaping goals: right-CVaR for seeking novel samples in the high-reward tail and left-CVaR for controlling worst-case samples in the low-reward tail. Unlike prior approaches that rely on non-linear optimization, we leverage the variational dual formulation of CVaR to decompose it into a decoupled two-stage procedure: a lightweight one-dimensional threshold optimization step, and a single entropy-regularized fine-tuning process via a specific pseudo-reward. This decomposition achieves CVaR fine-tuning efficiently with computational cost comparable to standard expected fine-tuning methods. We demonstrate the effectiveness of TFFT across illustrative experiments, high-dimensional text-to-image generation, and molecular design.

生成模型尾部控制流模型强化学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。