提出首个文本到图像模型选择框架,高效匹配最优微调模型。
Match & Choose: Model Selection Framework for Fine-tuning Text-to-Image Diffusion Models
- 构建匹配图,融合模型与数据特征预测微调效果
- 在32个数据集上准确选出最佳微调模型61.3%
- 适合希望省去全量微调试错的生成应用开发者
基于扩散和Transformer架构的文本到图像(T2I)模型发展迅速,通常在大规模语料库上预训练,并公开共享于HuggingFace等平台。用户可通过采用预训练T2I模型并在目标数据集上微调,构建AI应用,如媒体内容生成。尽管公开模型推动了技术普及,但用户面临新挑战:如何根据目标数据领域选择最适配的预训练模型?模型选择在分类任务中已有成熟方法,但在(预训练)T2I模型及其在目标域的表现预测方面仍缺乏研究。本文提出首个模型选择框架M&C,使用户无需对所有模型进行全量微调即可高效选择最优预训练模型。M&C核心为匹配图,包含可用模型和已分析数据集的节点,以及表示微调性能和数据相似性的模型-数据、数据-数据边。通过结合模型/数据特征与匹配图嵌入特征,构建预测模型,以预估在目标域微调后表现最佳的模型。我们在10个T2I模型和32个数据集上评估M&C,对比三种基线。结果表明,M&C在61.3%情况下成功预测出最佳微调模型,其余情况则选中表现相近的模型。
原文摘要 · Abstract (English)
Text-to-image (T2I) models based on diffusion and transformer architectures advance rapidly. They are often pretrained on large corpora, and openly shared on a model platform, such as HuggingFace. Users can then build up AI applications, e.g., generating media contents, by adopting pretrained T2I models and fine-tuning them on the target dataset. While public pretrained T2I models facilitate the democratization of the models, users face a new challenge: which model can be best fine-tuned based on the target data domain? Model selection is well addressed in classification tasks, but little is known in (pretrained) T2I models and their performance indication on the target domain. In this paper, we propose the first model selection framework, M&C, which enables users to efficiently choose a pretrained T2I model from a model platform without exhaustively fine-tuning them all on the target dataset. The core of M&C is a matching graph, which consists of: (i) nodes of available models and profiled datasets, and (ii) edges of model-data and data-data pairs capturing the fine-tuning performance and data similarity, respectively. We then build a model that, based on the inputs of model/data feature, and, critically, the graph embedding feature, extracted from the matching graph, predicts the model achieving the best quality after fine-tuning for the target domain. We evaluate M&C on choosing across ten T2I models for 32 datasets against three baselines. Our results show that M&C successfully predicts the best model for fine-tuning in 61.3% of the cases and a closely performing model for the rest.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。