arXiv:2412.12232cs.CVcs.AI2024-12中稿 · NeurIPS被引 3

用户只传一张图,系统就能自动找到最合适的生成模型。

You Only Submit One Image to Find the Most Suitable Generative Model

  • 用单张图像描述需求,通过三模块联合识别最佳生成模型。
  • 在大规模模型中实现超过80%的前4名识别准确率。
  • 适合需要快速匹配生成模型的创作者和开发者使用。

深度生成模型在图像生成方面取得了显著进展,各类模型平台(如Hugging Face、Civitai)允许开发者上传模型,用户下载使用。然而,这些平台缺乏先进的模型管理与识别机制,用户只能通过文本匹配或下载排序等方式搜索,难以高效找到最符合需求的模型。本文提出一种新型任务——生成模型识别(Generative Model Identification, GMI),旨在从大量候选模型中高效识别出最适配用户需求的模型。据我们所知,该问题尚未被研究。为此,本文提出一个包含三个核心模块的综合解决方案:基于加权缩减核均值嵌入(RKME)的框架,用于捕捉生成图像分布及图像与提示间的关系;预训练视觉语言模型以应对维度挑战;以及设计图像问询器解决跨模态问题。大量实验证明,该方法兼具高效性与有效性。例如,用户仅需提交一张示例图像,平台即可实现平均前4名识别准确率超过80%。

原文摘要 · Abstract (English)

Deep generative models have achieved promising results in image generation, and various generative model hubs, e.g., Hugging Face and Civitai, have been developed that enable model developers to upload models and users to download models. However, these model hubs lack advanced model management and identification mechanisms, resulting in users only searching for models through text matching, download sorting, etc., making it difficult to efficiently find the model that best meets user requirements. In this paper, we propose a novel setting called Generative Model Identification (GMI), which aims to enable the user to identify the most appropriate generative model(s) for the user's requirements from a large number of candidate models efficiently. To our best knowledge, it has not been studied yet. In this paper, we introduce a comprehensive solution consisting of three pivotal modules: a weighted Reduced Kernel Mean Embedding (RKME) framework for capturing the generated image distribution and the relationship between images and prompts, a pre-trained vision-language model aimed at addressing dimensionality challenges, and an image interrogator designed to tackle cross-modality issues. Extensive empirical results demonstrate the proposal is both efficient and effective. For example, users only need to submit a single example image to describe their requirements, and the model platform can achieve an average top-4 identification accuracy of more than 80%.

模型识别生成模型视觉语言单图输入

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。