用视觉语言模型指导低光图像增强,提升真实感与语义准确性
Adapting Large VLMs with Iterative and Manual Instructions for Generative Low-light Enhancement
- 通过正常光照图文描述生成语义先验,指导增强过程
- 迭代优化文本指令,使低光图像恢复更清晰自然
- 支持用户手动输入指令,实现个性化增强效果
现有低光图像增强方法多依赖预训练模型先验或低光输入,忽视了正常光照图像中的语义信息,导致复杂光照下效果受限。本文提出VLM-IMI框架,利用大视觉语言模型结合迭代与人工指令进行生成式低光增强。该框架包含两个分支:正常光照指令先验生成(NL-IPG)和指令感知光照增强扩散(IA-LED)。NL-IPG将期望的正常光照内容描述作为增强提示,实现语义引导的复原;IA-LED则融合NL-IPG生成的指令先验,引导扩散过程实现精准光照增强。为有效整合跨模态先验,引入可学习的指令先验融合模块,动态对齐并融合图像与文本特征,促进生成细节丰富且语义一致的结果。推理时因无真实正常光照图像,采用迭代指令策略逐步优化文本提示,持续提升视觉质量。同时,支持用户直接输入自定义指令,由LLM生成期望输出。在多种场景下的实验表明,VLM-IMI在感知质量和真实性上均优于现有最先进方法。代码已开源。
原文摘要 · Abstract (English)
Most existing low-light image enhancement (LLIE) methods rely on pre-trained model priors, low-light inputs, or both, while neglecting the semantic guidance available from normal-light images. This limitation hinders their effectiveness in complex lighting conditions. In this paper, we propose VLM-IMI, a framework that adapts large vision-language models with iterative and manual instructions for generative LLIE. VLM-IMI mainly contains two branches: Normal-Light Instruction Prior Generation (NL-IPG) and Instruction-aware Light Enhancement Diffusion (IA-LED). The NL-IPG incorporates textual descriptions of the desired normal-light content as enhancement cues, enabling semantically informed restoration. IA-LED incorporates instruction priors from the NL-IPG to guide the diffusion process, enabling precise illumination enhancement. To effectively integrate cross-modal priors, we introduce a learnable instruction prior fusion module, which dynamically aligns and fuses image and text features, promoting the generation of detailed and semantically coherent outputs. During inference, as the ground-truth normal-light images are not available, we propose an inference with an iterative instructions strategy to refine textual instructions, progressively improving visual quality. Our VLM-IMI also inherently supports manual instruction control by allowing users to directly input custom instructions into the LLM to generate user-expected outputs. Experiments across diverse scenarios demonstrate that VLM-IMI outperforms SOTA methods in terms of perception and realism. The source code is available at: https://github.com/sunxiaoran01/VLM-IMI.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。