用文字控制人像光影,实现创意级光照重置。
Text2Relight: Creative Portrait Relighting with Text Guidance
- 通过大模型生成多样化文本提示,驱动光照图像生成。
- 利用光场采集数据实现前后景光照精准重置,支持背景点光源迁移。
- 构建大规模合成数据集,提升文本到光照的泛化能力。
我们提出一种光照感知的图像编辑流程,给定一张人像图像和文本提示,可完成单图光照重制。模型同时调整前景与背景的光照与色彩,使其符合文本描述。由于文本描述在创造性上无边界,可涵盖温度、情绪、时间等感官特征,但现有数据缺乏大规模文本-光照配对,导致当前模型难以泛化至光照特定任务。为此,我们设计新型数据合成流程:首先用大语言模型(如ChatGPT)生成多样且富有创意的文本提示;再通过文本引导的图像生成模型生成匹配的光照图像;以该光照图为条件,基于光场系统采集的OLAT(One-Light-at-a-Time)图像,对前/背景进行图像级光照重制。尤其对于背景,将光照图像表示为点光源集合,并迁移至其他背景图像。最后,采用生成式扩散模型学习合成的海量数据,辅以人物美化、光照定位等辅助任务增强,建立文本与光照潜在分布间的关联,实现文本引导的人像光照重制。
原文摘要 · Abstract (English)
We present a lighting-aware image editing pipeline that, given a portrait image and a text prompt, performs single image relighting. Our model modifies the lighting and color of both the foreground and background to align with the provided text description. The unbounded nature in creativeness of a text allows us to describe the lighting of a scene with any sensory features including temperature, emotion, smell, time, and so on. However, the modeling of such mapping between the unbounded text and lighting is extremely challenging due to the lack of dataset where there exists no scalable data that provides large pairs of text and relighting, and therefore, current text-driven image editing models does not generalize to lighting-specific use cases. We overcome this problem by introducing a novel data synthesis pipeline: First, diverse and creative text prompts that describe the scenes with various lighting are automatically generated under a crafted hierarchy using a large language model (*e.g.,* ChatGPT). A text-guided image generation model creates a lighting image that best matches the text. As a condition of the lighting images, we perform image-based relighting for both foreground and background using a single portrait image or a set of OLAT (One-Light-at-A-Time) images captured from lightstage system. Particularly for the background relighting, we represent the lighting image as a set of point lights and transfer them to other background images. A generative diffusion model learns the synthesized large-scale data with auxiliary task augmentation (*e.g.,* portrait delighting and light positioning) to correlate the latent text and lighting distribution for text-guided portrait relighting.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。