用自回归模型统一实现多种图像生成与编辑任务
EditAR: Unified Conditional Generation with Autoregressive Models
- 基于自回归框架，统一处理图像编辑、深度转图像等任务
- 在多个基准上表现媲美专用先进模型，支持文本与图像对齐
- 适合需要统一生成框架的研究者与开发者
可控图像生成与编辑的最新进展主要依赖于扩散模型。尽管扩散模型在特定任务中表现优异，但构建统一模型仍具挑战。相比之下，自回归模型天然具备统一的令牌化表示，更易构建单一基础模型。本文提出 EditAR，一种统一的自回归框架，支持多种条件图像生成任务，如图像编辑、深度图转图像、边缘图转图像、分割图转图像。模型以图像和指令为输入，采用标准的下一个令牌预测方式生成编辑后的图像令牌。为增强文本到图像的对齐，我们进一步提出将基础模型的知识蒸馏进自回归建模过程。在多个基准上的评估显示，其性能在各类任务中均达到或接近当前最优任务专用方法。项目页面：https://jitengmu.github.io/EditAR/
原文摘要 · Abstract (English)
Recent progress in controllable image generation and editing is largely driven by diffusion-based methods. Although diffusion models perform exceptionally well in specific tasks with tailored designs, establishing a unified model is still challenging. In contrast, autoregressive models inherently feature a unified tokenized representation, which simplifies the creation of a single foundational model for various tasks. In this work, we propose EditAR, a single unified autoregressive framework for a variety of conditional image generation tasks, e.g., image editing, depth-to-image, edge-to-image, segmentation-to-image. The model takes both images and instructions as inputs, and predicts the edited images tokens in a vanilla next-token paradigm. To enhance the text-to-image alignment, we further propose to distill the knowledge from foundation models into the autoregressive modeling process. We evaluate its effectiveness across diverse tasks on established benchmarks, showing competitive performance to various state-of-the-art task-specific methods. Project page: https://jitengmu.github.io/EditAR/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。