通过分区域混合大模型小模型，加速文生图生成。
HybridStitch: Pixel and Timestep Level Model Stitching for Diffusion Acceleration
- 将图像分为易渲染区和复杂区，分用大小模型处理
- 在Stable Diffusion 3上实现1.83倍加速
- 适合需要快速生成图像的实时应用
扩散模型在文生图（T2I）生成中表现出色，但大型模型（数十亿参数）计算开销巨大。已有方法仅在部分时间步替换为小模型，忽略单个时间步内计算需求差异。本文提出HybridStitch，将生成过程类比为编辑：将图像分为易渲染区域与复杂区域，分别使用小模型生成粗略草图，大模型精细修正复杂区域。该混合阶段联合利用大、小模型，实现高效协同。实验表明，HybridStitch在Stable Diffusion 3上达到1.83×加速，优于现有所有混合模型方法。
原文摘要 · Abstract (English)
Diffusion models have demonstrated a remarkable ability in Text-to-Image (T2I) generation applications. Despite the advanced generation output, they suffer from heavy computation overhead, especially for large models that contain tens of billions of parameters. Prior work has illustrated that replacing part of the denoising steps with a smaller model still maintains the generation quality. However, these methods only focus on saving computation for some timesteps, ignoring the difference in compute demand within one timestep. In this work, we propose HybridStitch, a new T2I generation paradigm that treats generation like editing. Specifically, we introduce a hybrid stage that jointly incorporates both the large model and the small model. HybridStitch separates the entire image into two regions: one that is relatively easy to render, enabling an early transition to the smaller model, and another that is more complex and therefore requires refinement by the large model. HybridStitch employs the small model to construct a coarse sketch while exploiting the large model to edit and refine the complex regions. According to our evaluation, HybridStitch achieves 1.83$\times$ speedup on Stable Diffusion 3, which is faster than all existing mixture of model methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。