仅用一张照片-草图对，就能生成逼真草图，无需大量训练数据。
One-shot Face Sketch Synthesis in the Wild via Generative Diffusion Prior and Instruction Tuning
- 基于扩散模型，通过文本指令优化实现单次样本生成。
- 在400对真实场景数据上测试，生成草图与目标风格高度一致。
- 适合缺乏标注数据的实战场景，尤其适用于个性化草图生成。
人脸草图合成旨在将人脸照片转换为草图。现有方法多依赖大规模照片-草图成对数据训练，但面临数据稀缺和人工成本高的问题。当训练数据不足时，生成性能显著下降。本文提出一种基于扩散模型的单次（one-shot）人脸草图合成方法，通过人脸照片-草图图像对优化扩散模型的文本指令，利用梯度优化得到的指令进行推理。为更真实评估方法有效性，我们构建新基准One-shot Face Sketch Dataset (OS-Sketch)，包含400对照片-草图，涵盖不同风格草图、背景、年龄、性别、表情、光照等变化。实验中每轮仅使用一对图像训练，其余用于推理。大量实验表明，该方法可在单次样本条件下生成逼真且风格一致的草图，相比现有方法更具便利性和适用性。数据集已开源：https://github.com/HanWu3125/OS-Sketch。
原文摘要 · Abstract (English)
Face sketch synthesis is a technique aimed at converting face photos into sketches. Existing face sketch synthesis research mainly relies on training with numerous photo-sketch sample pairs from existing datasets. However, these large-scale discriminative learning methods will have to face problems such as data scarcity and high human labor costs. Once the training data becomes scarce, their generative performance significantly degrades. In this paper, we propose a one-shot face sketch synthesis method based on diffusion models. We optimize text instructions on a diffusion model using face photo-sketch image pairs. Then, the instructions derived through gradient-based optimization are used for inference. To simulate real-world scenarios more accurately and evaluate method effectiveness more comprehensively, we introduce a new benchmark named One-shot Face Sketch Dataset (OS-Sketch). The benchmark consists of 400 pairs of face photo-sketch images, including sketches with different styles and photos with different backgrounds, ages, sexes, expressions, illumination, etc. For a solid out-of-distribution evaluation, we select only one pair of images for training at each time, with the rest used for inference. Extensive experiments demonstrate that the proposed method can convert various photos into realistic and highly consistent sketches in a one-shot context. Compared to other methods, our approach offers greater convenience and broader applicability. The dataset will be available at: https://github.com/HanWu3125/OS-Sketch
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。