用轻量化模型实现在消费级显卡上视频级文本编辑生成，突破传统瓶颈。
Inverting the Streaming-Diffusion Bottleneck: Video-Rate MLLM-Conditioned Edit Diffusion on a Consumer GPU
- 通过极简蒸馏压缩扩散模型，将性能瓶颈从去噪器转移到多模态大模型编码器。
- 在RTX 3090 Ti上实现27-30帧/秒，RTX 5090可达74帧/秒，支持长视频流式生成。
- 适合关注视频生成系统优化与推理效率的开发者和研究者。
对扩散U-Net进行激进蒸馏，反转了实时文生图流水线的逐帧瓶颈：当去噪器变为4步或1步蒸馏学生模型后，文本编码器成为关键路径。该现象在视觉感知编辑扩散中尤为突出，其中编码器为多模态大语言模型（MLLM）。我们研究了一个0.39B参数的蒸馏编辑U-Net与一个2.13B参数的MLLM文本编码器（Qwen3-VL），并提出一种基于三项机制的流式流水线，使编码器不成为去噪器的关键路径：异步侧流/主流CUDA流水线结合批处理文本编码器分摊、可编译的ControlNet-LLLite重构（将整个U-Net+适配器堆叠融合为单一图结构）、以及周期性条件刷新调度配合钩子子集，分摊每帧条件成本。在单张消费级RTX 3090 Ti上，512x512分辨率下持续保持27-30 fps，480帧运行；同条件下，稳态吞吐量在RTX 4090达55 fps，RTX 5090达74 fps。结果表明，一旦蒸馏足够激进，进一步提升来自编码器侧系统优化，而非继续压缩去噪器——这与现有流式扩散研究优化方向相反。报告的是视频级流式吞吐率，非交互式低延迟。数字与相同架构的StreamDiffusion重跑对比，非优越性声明。发布的油画风格适配器在19个未见的DAVIS-2017序列及15个非DAVIS来源共七类视频中泛化成功，提示级风格泛化能力有限且单独报告。
原文摘要 · Abstract (English)
Aggressive distillation of the diffusion U-Net inverts the per-frame bottleneck of real-time text-to-image pipelines: once the denoiser is a 4-step or 1-step distilled student, the text encoder becomes the critical path. This inversion is most acute in vision-aware edit diffusion, where the encoder is a multimodal large language model (MLLM). We study a 0.39B distilled edit U-Net paired with a 2.13B MLLM text encoder (Qwen3-VL) and present a streaming pipeline for this regime built on three mechanisms that keep the encoder off the denoiser's critical path rather than shrinking the encoder: asymmetric side-stream / main-stream CUDA pipelining with batched text-encoder amortisation, a compile-friendly ControlNet-LLLite reformulation that folds the whole U-Net + adapter stack into one fused graph, and a periodic conditioning-refresh schedule with a hook subset that amortises the per-frame conditioning cost. On a single consumer RTX 3090 Ti at 512x512 this sustains 27-30 fps over a 480-frame run; at the same operating point steady-state throughput scales to 55 fps on RTX 4090 and 74 fps on RTX 5090. This shows that once distillation is aggressive enough, further gains come from encoder-side systems work rather than further denoiser compression -- the opposite lever from the one the streaming-diffusion literature has optimised to date. We report video-rate streaming throughput, not interactive low latency, and locate our numbers against same-stack StreamDiffusion re-runs as systems context, not a superiority claim. The released oil-painting adapter generalises within in-clip noise to 19 unused DAVIS-2017 sequences and 15 non-DAVIS clips from seven sources; prompt-level generalisation to unseen styles is bounded and reported separately.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。