arXiv:2409.18128cs.CV2024-09NeurIPS被引 5

FlowTurbo让流模型生成图像更快更准,实现每秒100帧的实时生成。

FlowTurbo: Towards Real-time Flow-Based Image Generation with Velocity Refiner

  • 用轻量速度修正器预测采样稳定期的运动方向,提升效率
  • 在ImageNet上达2.12 FID,每图仅需100毫秒,突破实时生成极限
  • 适用于图像编辑、修复等任务,可适配多种流模型

基于扩散模型在视觉生成中的成功,流模型作为另一类主流生成模型重新崛起,在视觉质量与推理速度方面达到竞争力或更优表现。通过流匹配学习速度场,流模型通常具有更直的采样轨迹,有利于采样过程。然而,与扩散模型已有成熟快速采样器不同,流模型的高效采样研究仍较少。本文提出名为FlowTurbo的框架,在保持采样质量的同时加速流模型采样。核心观察是:流模型采样过程中速度预测器输出趋于稳定,可借助轻量级速度修正器进行估算。此外,引入伪校正器与样本感知编译技术进一步降低推理时间。由于不改变多步采样范式,FlowTurbo可有效应用于图像编辑、图像修复等多种任务。将其集成至多个流模型后,类别条件生成加速率达53.1%~58.3%,文本到图像生成加速率达29.8%~38.5%。特别地,FlowTurbo在ImageNet上实现2.12 FID(100 ms/img)和3.93 FID(38 ms/img),达成实时图像生成,并建立新最优性能。代码已公开于https://github.com/shiml20/FlowTurbo。

原文摘要 · Abstract (English)

Building on the success of diffusion models in visual generation, flow-based models reemerge as another prominent family of generative models that have achieved competitive or better performance in terms of both visual quality and inference speed. By learning the velocity field through flow-matching, flow-based models tend to produce a straighter sampling trajectory, which is advantageous during the sampling process. However, unlike diffusion models for which fast samplers are well-developed, efficient sampling of flow-based generative models has been rarely explored. In this paper, we propose a framework called FlowTurbo to accelerate the sampling of flow-based models while still enhancing the sampling quality. Our primary observation is that the velocity predictor's outputs in the flow-based models will become stable during the sampling, enabling the estimation of velocity via a lightweight velocity refiner. Additionally, we introduce several techniques including a pseudo corrector and sample-aware compilation to further reduce inference time. Since FlowTurbo does not change the multi-step sampling paradigm, it can be effectively applied for various tasks such as image editing, inpainting, etc. By integrating FlowTurbo into different flow-based models, we obtain an acceleration ratio of 53.1%$\sim$58.3% on class-conditional generation and 29.8%$\sim$38.5% on text-to-image generation. Notably, FlowTurbo reaches an FID of 2.12 on ImageNet with 100 (ms / img) and FID of 3.93 with 38 (ms / img), achieving the real-time image generation and establishing the new state-of-the-art. Code is available at https://github.com/shiml20/FlowTurbo.

流模型实时生成速度修正图像生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。