统一控制扩散模型的插件框架,支持多种生成任务组合使用。
Diffusion Templates: A Unified Plugin Framework for Controllable Diffusion

- 将控制能力抽象为模板,通过标准接口注入基础扩散模型。
- 支持KV缓存、LoRA等异构控制模块共存与组合,兼容不同骨干网络。
- 涵盖结构控制、调色、修复等10+任务,适合快速扩展新功能的研究者。
可控扩散方法显著拓展了扩散模型的实际应用,但通常以孤立、依赖特定骨干网络的系统形式开发,训练流程、参数格式和运行时钩子不兼容。这种碎片化导致跨任务复用基础设施困难,难以在不同骨干间迁移能力,也无法在一个生成流程中组合多个控制。本文提出Diffusion Templates,一个统一且开源的插件框架,将基础模型推理与可控能力注入解耦。框架围绕三个组件构建:模板模型将任意任务输入映射为中间能力表示;模板缓存作为标准化的能力注入接口;模板流水线负责加载、合并并注入一个或多个模板缓存到基础扩散运行时。由于接口在系统层面定义,而非绑定特定控制架构,各类能力载体(如KV-Cache、LoRA)可在同一抽象下被支持。基于此设计,我们构建了一个涵盖结构控制、亮度调整、色彩调节、图像编辑、超分辨率、锐度增强、审美对齐、内容参考、局部修复和年龄控制等10余项任务的模型库。案例研究证明,Diffusion Templates可统一多种可控生成任务,同时保持模块性、可组合性和对快速演进的扩散骨干网络的实用性扩展能力。所有资源(代码、模型、数据集)将开源。
原文摘要 · Abstract (English)
Controllable diffusion methods have substantially expanded the practical utility of diffusion models, but they are typically developed as isolated, backbone-specific systems with incompatible training pipelines, parameter formats, and runtime hooks. This fragmentation makes it difficult to reuse infrastructure across tasks, transfer capabilities across backbones, or compose multiple controls within a single generation pipeline. We present Diffusion Templates, a unified and open plugin framework that decouples base-model inference from controllable capability injection. The framework is organized around three components: Template models that map arbitrary task-specific inputs to an intermediate capability representation, a Template cache that functions as a standardized interface for capability injection, and a Template pipeline that loads, merges, and injects one or more Template caches into the base diffusion runtime. Because the interface is defined at the systems level rather than tied to a specific control architecture, heterogeneous capability carriers such as KV-Cache and LoRA can be supported under the same abstraction. Based on this design, we build a diverse model zoo spanning structural control, brightness adjustment, color adjustment, image editing, super-resolution, sharpness enhancement, aesthetic alignment, content reference, local inpainting, and age control. These case studies show that Diffusion Templates can unify a broad range of controllable generation tasks while preserving modularity, composability, and practical extensibility across rapidly evolving diffusion backbones. All resources will be open sourced, including code, models, and datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。