让视觉控制更精准:通过特征对齐提升生成模型的训练效率和可控性。
LISA: Likelihood Score Alignment for Visual-condition Controllable Generation

- 用近似似然得分引导侧支网络特征,实现条件控制对齐。
- 训练收敛速度加快,生成质量提升,特征更解耦且无额外推理开销。
- 适用于图像/视频生成,尤其适合追求高效可控生成的研究者。
主流的双分支范式——即训练一个侧支网络编码视觉条件,并将中间层特征融合到冻结的预训练主网络中——在视觉条件可控生成任务中表现卓越。然而,侧支网络的作用及其训练效率仍缺乏深入研究。本文从基于分数的生成建模视角重新审视该范式:1)主网络通过提供无条件分数保持视觉感知质量;2)侧支网络通过隐式贡献似然分数实现条件控制。受此启发,我们提出 LIkelihood Score Alignment(LISA),一种有效正则化方法,显式对齐侧支网络中间特征与近似似然分数。具体地,从侧支网络指定层提取特征,通过轻量解码器投影至分数潜在空间;构造近似似然分数目标,计算解码输出与目标之间的距离作为额外正则化损失;联合优化侧支网络与解码器,同时使用标准扩散损失和正则化损失。在多种图像/视频任务、架构及扩散/流模型上实验表明,LISA不仅能持续加速训练收敛并提升最终生成效果,还能使侧支网络特征更具解耦性,且训练成本几乎不变,推理零新增开销。
原文摘要 · Abstract (English)
The prevalent dual-branch paradigm, i.e., training a side network to encode visual conditions and fusing its intermediate-layer features to a frozen pretrained main network, has shown remarkable success in visual-condition controllable generation. Despite its widespread adoption, the role of the side branch and its training efficiency remain underexplored. In this paper, we first revisit this mainstream paradigm through the lens of score-based generative modeling: 1) The main network preserves visual perceptual quality by providing a prior unconditional score. 2) The side network steers conditional control by implicitly contributing a likelihood score. Guided by this perspective, we propose LIkelihood Score Alignment (LISA), an effective regularization method that explicitly aligns the intermediate feature of the side network with an approximated likelihood score. Specifically, we first hook features from a designated layer of the side network and project them into the score latent space by a lightweight decoder. Then, we construct an approximated likelihood score target and calculate the distance between the decoder's output and this target as an additional regularization loss. Finally, we jointly optimize the side network and decoder with both standard diffusion loss and our regularization loss. Experiments across various image/video tasks, architectures, and diffusion/flow models demonstrated that LISA can not only consistently accelerate the training convergence and improve final synthetic results, but also encourage the side network's features to be more disentangled for conditional modeling with negligible additional training cost and zero extra inference cost.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。