通过轻量控制接口实现扩散模型内容与风格的平滑调节
SteeringDiffusion: A Bottlenecked Activation Control Interface for Diffusion Models
- 用小规模提示条件潜码调节基础模型，不改动主干网络
- 单个标量即可在推理时连续调整内容与风格比例
- 比LoRA更可控稳定，适合需要精准调节的生成任务
我们提出SteeringDiffusion，一种基于瓶颈激活控制的扩散模型接口，可在内容与风格之间提供平滑、单调且可实时调节的控制表面。方法保持U-Net主干冻结，学习一个小型、提示条件化的潜码，并映射为FiLM/AdaGN风格的调制参数。零初始化设计确保在零尺度下与原模型完全等价，时间步感知门控仅在去噪后期生效。推理时只需一个标量即可连续遍历控制面，无需重训练。在Stable Diffusion 1.5和SDXL上覆盖多种艺术风格的实验表明，该方法能实现平滑连续的内容-风格权衡。在相同参数预算下，其可控性与稳定性优于LoRA，而ControlNet和秩-1适配器无法提供类似控制面。我们还引入基于DDIM反演的稳定性诊断工具，作为事后轨迹探针，揭示干预幅度与稳定性之间的强相关性。这些结果表明，Steering Bottlenecked Explicit Control (S-BEC)是适用于冻结扩散主干的实用通用控制接口。
原文摘要 · Abstract (English)
We introduce SteeringDiffusion, a bottlenecked activation-level control interface for diffusion models that exposes a smooth, monotonic, and runtime-adjustable control surface over the content--style trade-off. Our method keeps the U-Net backbone frozen and learns a small, prompt-conditioned latent code projected to FiLM/AdaGN-style modulation parameters. A zero-initialized design guarantees exact equivalence to the base model at zero scale, while timestep-aware gating restricts modulation to later denoising stages. A single scalar at inference continuously traverses the control surface without retraining. Across experiments on Stable Diffusion~1.5 and SDXL covering multiple artistic styles, we show that SteeringDiffusion produces smooth and monotonic content--style trade-offs. Under matched parameter budgets, it outperforms LoRA in controllability and stability, while ControlNet and rank-1 adapters do not expose a comparable control surface. We further introduce an inversion-stability diagnostic based on DDIM inversion, used as a post-hoc trajectory probe, which reveals strong correlations with intervention magnitude. These results position \emph{Steering Bottlenecked Explicit Control (S-BEC)} as a practical, general-purpose control interface for frozen diffusion backbones.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。