用大模型搭建可组合的图像编辑积木,让指令编辑更灵活通用。
Lego-Edit: A General Image Editing Framework with Model-Level Bricks and MLLM Builder
- 构建模型级工具库,支持细粒度编辑动作组合。
- 三阶段强化学习提升大模型对开放域指令的泛化能力。
- 新工具加入无需微调,适合实际场景快速部署。
基于指令的图像编辑因与用户直接交互而受到广泛关注。然而,真实用户指令种类繁多,现有方法往往难以泛化到训练数据之外的指令,限制了实际应用。为此,我们提出Lego-Edit,利用多模态大语言模型(MLLM)的泛化能力,组织一套模型级编辑工具以应对该挑战。Lego-Edit包含两个关键设计:(1) 一个由多种在有限数据上高效训练的模型及若干图像操作函数组成的模型级工具包,使MLLM能够实现细粒度编辑动作的组合;(2) 一种三阶段渐进式强化学习方法,通过在未标注的开放域指令上进行反馈训练MLLM,赋予其处理真实指令的通用推理能力。实验表明,Lego-Edit在GEdit-Bench和ImgBench上达到当前最优性能,展现出对开放域指令的强推理能力,并可在不额外微调的情况下使用新引入的编辑工具。代码已开源:https://github.com/xiaomi-research/lego-edit。
原文摘要 · Abstract (English)
Instruction-based image editing has garnered significant attention due to its direct interaction with users. However, real-world user instructions are immensely diverse, and existing methods often fail to generalize effectively to instructions outside their training domain, limiting their practical application. To address this, we propose Lego-Edit, which leverages the generalization capability of Multi-modal Large Language Model (MLLM) to organize a suite of model-level editing tools to tackle this challenge. Lego-Edit incorporates two key designs: (1) a model-level toolkit comprising diverse models efficiently trained on limited data and several image manipulation functions, enabling fine-grained composition of editing actions by the MLLM; and (2) a three-stage progressive reinforcement learning approach that uses feedback on unannotated, open-domain instructions to train the MLLM, equipping it with generalized reasoning capabilities for handling real-world instructions. Experiments demonstrate that Lego-Edit achieves state-of-the-art performance on GEdit-Bench and ImgBench. It exhibits robust reasoning capabilities for open-domain instructions and can utilize newly introduced editing tools without additional fine-tuning. Code is available: https://github.com/xiaomi-research/lego-edit.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。