统一视觉理解与生成的像素流分块编码器,兼顾语义与细节重建。
UniFlow: A Unified Pixel Flow Tokenizer for Visual Understanding and Generation
- 通过层级自蒸馏适配预训练编码器,融合高层语义与低层像素信息。
- 采用轻量级分块像素流解码器,实现高保真图像重建,提升训练效率。
- 在13项任务中表现均衡,适合多模态建模与生成场景研究者使用。
Tokenizer 是视觉理解与生成的关键组件。为实现通用建模目标,近期研究致力于开发统一的 tokenizer。然而,现有方法在理解与生成间存在显著性能权衡,源于高层语义抽象与底层像素重构之间的固有冲突。为此,我们提出一种通用且统一的 tokenizer——UniFlow,通过将任意视觉编码器与简洁的重建解码器灵活结合。具体而言,引入应用于预训练视觉编码器的层级自适应自蒸馏机制,使 UniFlow 同时继承强语义特征以支持理解,并灵活适应细粒度细节以支持生成。此外,提出轻量级分块像素流解码器,通过建模从噪声状态到分块像素域的条件流,高效实现高保真像素重构。利用语义特征作为解码器的视觉条件,有效缓解理解与生成间的训练冲突。同时,分块学习策略简化数据分布,提升训练效率。在涵盖7类主流视觉理解与生成任务的13个挑战性基准上进行的大量实验表明,UniFlow 实现了双赢效果。例如,我们的 7B UniFlow-XL 在平均理解基准上超越 14B TokenFlow-XL 6.05%,在图像重建与生成方面也表现优异,分别在 rFID 和 gFID(无引导)上超越 UniTok 0.15 与 0.09。
原文摘要 · Abstract (English)
Tokenizer is a crucial component for both visual understanding and generation. To advance toward the ultimate goal of universal modeling, recent research has focused on developing a unified tokenizer. However, existing tokenizers face a significant performance trade-off between understanding and generation, stemming from the inherent conflict between high-level semantic abstraction and low-level pixel reconstruction. To tackle this challenge, we propose a generic and unified tokenizer, namely UniFlow, by flexibly adapting any visual encoder with a concise reconstruction decoder. Specifically, we introduce layer-wise adaptive self-distillation applied to the well-pretrained visual encoders, which enables UniFlow to simultaneously inherit the strong semantic features for visual understanding and flexibly adapt to model fine-grained details for visual generation. Moreover, we propose a lightweight patch-wise pixel flow decoder, which efficiently achieves high-fidelity pixel reconstruction by modeling a conditional flow from the noisy state back to the patch-wise pixel domain. By leveraging the semantic features as visual conditions for the decoder, we effectively alleviate the training conflicts between understanding and generation. Furthermore, the patch-wise learning strategy simplifies the data distribution, thereby improving training efficiency. Extensive experiments across 13 challenging benchmarks spanning 7 widely studied visual understanding and generation tasks demonstrate that UniFlow achieves a win-win outcome. For instance, our 7B UniFlow-XL not only surpasses the 14B TokenFlow-XL by 6.05% on average understanding benchmarks, but also achieves a competitive results in both visual reconstruction and generation, surpassing UniTok by 0.15 in rFID and 0.09 in gFID (without guidance), respectively.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。