用知识蒸馏融合多个图像生成模型，实现可控风格生成。
DMM: Building a Versatile Image Generation Model via Distillation-Based Model Merging
- 基于得分蒸馏的模型合并方法，通过风格向量控制生成样式。
- 可将多模型知识压缩为单一模型，支持任意风格可控生成。
- 适合需要多功能图像生成能力的研究者与开发者使用。
文本到图像（T2I）生成模型的成功催生了大量在不同专用数据集上微调的模型检查点。这种海量专用模型带来了参数冗余和存储成本高的问题，亟需有效方法将多种强大模型的能力整合为单一模型。现有模型合并方法通常在参数空间中采用静态线性插值，但忽略了T2I任务中不同模型覆盖多样风格的特点，易导致合并后模型风格不兼容、混乱。为此，我们提出一种可风格提示的图像生成流程，能精准控制生成任意风格图像。基于此，我们设计了基于得分蒸馏的模型合并范式（DMM），将多个模型压缩为一个通用的T2I模型。同时，我们重新思考并重构了T2I生成场景下的模型合并任务，提出了新的合并目标与评估协议。实验表明，DMM能紧凑地重组多个教师模型的知识，实现可控的任意风格生成。
原文摘要 · Abstract (English)
The success of text-to-image (T2I) generation models has spurred a proliferation of numerous model checkpoints fine-tuned from the same base model on various specialized datasets. This overwhelming specialized model production introduces new challenges for high parameter redundancy and huge storage cost, thereby necessitating the development of effective methods to consolidate and unify the capabilities of diverse powerful models into a single one. A common practice in model merging adopts static linear interpolation in the parameter space to achieve the goal of style mixing. However, it neglects the features of T2I generation task that numerous distinct models cover sundry styles which may lead to incompatibility and confusion in the merged model. To address this issue, we introduce a style-promptable image generation pipeline which can accurately generate arbitrary-style images under the control of style vectors. Based on this design, we propose the score distillation based model merging paradigm (DMM), compressing multiple models into a single versatile T2I model. Moreover, we rethink and reformulate the model merging task in the context of T2I generation, by presenting new merging goals and evaluation protocols. Our experiments demonstrate that DMM can compactly reorganize the knowledge from multiple teacher models and achieve controllable arbitrary-style generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。