通过智能分区与调度,让扩散模型生成更快更省资源。
FlashDiff: Efficient Regional Execution and Scheduling for Diffusion Model Serving

- 按潜在空间区域动态执行,只处理需要优化的部分。
- 实测生成延迟降低30%-97%,吞吐量提升1.2-2.2倍。
- 适合部署图像、视频、音频生成服务的工程师参考。
扩散模型已成为现代图像、视频和音频生成的核心架构,但其高效服务仍具挑战。与自回归解码不同,扩散推理需在多个去噪步骤中反复更新高维时空潜在表示,全区域执行导致延迟高、吞吐受限。现有多GPU并行方法虽可减少单步计算,但常引入大量激活数据交换开销,通信成本抵消甚至超过并行收益。本文提出FlashDiff,一种基于自适应区域执行与调度的扩散模型服务系统。其核心观察为:不同潜在区域在去噪过程中的收敛速度不一,相邻步骤间存在强时序相关性。FlashDiff利用此特性,仅选择需进一步优化的区域执行,并将释放出的算力重新分配给并发请求。系统包含三项机制:首先,基于早期注意力信号将潜在表示分解为语义一致的执行区域,暴露细粒度并行性;其次,采用轻量级运行时控制器估计区域活跃度,在进一步优化对输出质量影响有限时跳过低效更新;第三,设计亲和性感知在线调度器,协同放置依赖区域,均衡各GPU负载,并复用回收算力以提升服务效率。在真实世界图像、视频和音频工作负载上,FlashDiff实现端到端服务延迟降低30%-97%,吞吐量提升1.2-2.2倍。
原文摘要 · Abstract (English)
Diffusion models have become the central backbone for modern image, video, and audio generation, but their efficient service remains a challenge. Unlike autoregressive decoding, diffusion inference repeatedly updates high-dimensional spatial or temporal latents over many denoising steps. This all-region execution pattern makes generation latency high and limits serving throughput. Existing multi-GPU parallelization methods can reduce per-step computation, but often introduce substantial activation exchange overhead, causing communication to offset or even outweigh the benefits of parallel execution. This paper presents FlashDiff, a diffusion serving system that improves inference efficiency through adaptive regional execution and scheduling. FlashDiff is based on the observation that diffusion refinement is not uniform across latent regions or denoising steps: different regions often stabilize at different rates, while neighboring steps exhibit strong temporal correlation. FlashDiff leverages these properties to selectively execute only regions that require further refinement and to reallocate the resulting compute slack across concurrent serving requests. FlashDiff consists of three mechanisms. First, it decomposes the latent representation into coherent execution regions using early-stage attention signals, preserving semantic structure while exposing fine-grained parallelism. Second, it uses a lightweight runtime controller to estimate region activity and bypass low-impact updates when further refinement is unlikely to affect output quality. Third, it applies an affinity-aware online scheduler that co-locates dependent regions, balances residual load across GPUs, and reuses reclaimed compute capacity to improve serving efficiency. Across real-world image, video, and audio workloads, FlashDiff reduces end-to-end serving latency by 30-97% and improves throughput by 1.2-2.2x.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。