通过可插拔机制实现图文生成内容的多维度可控与可解释，兼顾公平安全。
Plug-and-Play Interpretable Responsible Text-to-Image Generation via Dual-Space Multi-facet Concept Control
- 用外部插件学习可解释的责任概念复合空间，不修改原模型。
- 在文本嵌入和扩散潜空间同时控制，实现多维度责任约束。
- 无需训练原模型，适合需要快速部署负责任生成的场景。
文本到图像（T2I）模型的伦理问题要求对生成内容进行全面控制。现有方法虽关注生成内容的公平性与安全性（如无暴力、无不良信息），但通常仅分别处理责任概念，缺乏可解释性，且常需修改原始模型，影响性能。本文提出一种新方法，通过外部可插拔机制，将目标T2I流水线知识蒸馏至一个可解释的复合责任空间，以条件化生成。该方法结合知识蒸馏与概念去相关技术，在推理阶段利用学习到的空间调节生成内容。典型T2I流水线提供两个插件点：文本嵌入空间与扩散模型潜空间。我们分别设计模块，验证了方法在多种强基线上的有效性。
原文摘要 · Abstract (English)
Ethical issues around text-to-image (T2I) models demand a comprehensive control over the generative content. Existing techniques addressing these issues for responsible T2I models aim for the generated content to be fair and safe (non-violent/explicit). However, these methods remain bounded to handling the facets of responsibility concepts individually, while also lacking in interpretability. Moreover, they often require alteration to the original model, which compromises the model performance. In this work, we propose a unique technique to enable responsible T2I generation by simultaneously accounting for an extensive range of concepts for fair and safe content generation in a scalable manner. The key idea is to distill the target T2I pipeline with an external plug-and-play mechanism that learns an interpretable composite responsible space for the desired concepts, conditioned on the target T2I pipeline. We use knowledge distillation and concept whitening to enable this. At inference, the learned space is utilized to modulate the generative content. A typical T2I pipeline presents two plug-in points for our approach, namely; the text embedding space and the diffusion model latent space. We develop modules for both points and show the effectiveness of our approach with a range of strong results.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。