arXiv:2603.15228cs.CV2026-03被引 1

提出统一生成与理解的视觉模型,通过渐进式特征学习提升性能。

HYDRA: Unifying Multi-modal Generation and Understanding via Representation-Harmonized Tokenization

  • 设计生成-语义瓶颈,分阶段融合生成与理解所需特征
  • 在重建任务中实现rFID 0.08,生成评测达0.86以上
  • 适合需要兼顾生成与理解的多模态应用开发者

统一多模态模型难以弥合视觉理解所需的抽象表示与生成所需细节之间的根本差距。现有方法通常通过解耦编码器、堆叠表示编码器于VAE之上或使用离散量化来妥协,但常破坏信息连贯性并引发优化冲突。为此,我们提出HYDRA-TOK,一种基于纯ViT架构的表示谐化模型,其核心思想是视觉建模应从生成走向理解。该模型将标准骨干网络重构为渐进式学习器,从保留结构的Gen-ViT过渡到语义编码的Sem-ViT。关键在于生成-语义瓶颈(GSB):先将特征压缩至低维空间以过滤噪声,保障稳健合成;再恢复维度以支持复杂语义理解。在此基础上,构建了原生统一框架HYDRA,将感知与生成整合于单一参数空间。大量实验表明,HYDRA达到新基准:视觉重建任务中rFID达0.08,生成评测中GenEval得分为0.86,DPG-Bench为86.4,WISE为0.53;同时在八项理解任务上平均超越前代原生统一模型10.0分。

原文摘要 · Abstract (English)

Unified Multimodal Models struggle to bridge the fundamental gap between the abstract representations needed for visual understanding and the detailed primitives required for generation. Existing approaches typically compromise by employing decoupled encoders, stacking representation encoder atop VAEs, or utilizing discrete quantization. However, these methods often disrupt information coherence and lead to optimization conflicts. To this end, we introduce HYDRA-TOK, a representation-harmonized pure ViT in the insight that visual modeling should evolve from generation to understanding. HYDRA-TOK reformulates the standard backbone into a progressive learner that transitions from a Gen-ViT, which captures structure-preserving primitives, to a Sem-ViT for semantic encoding. Crucially, this transition is mediated by a Generation-Semantic Bottleneck (GSB), which compresses features into a low-dimensional space to filter noise for robust synthesis, then restores dimensionality to empower complex semantic comprehension. Built upon this foundation, we present HYDRA, a native unified framework integrating perception and generation within a single parameter space. Extensive experiments establish HYDRA as a new state-of-the-art. It sets a benchmark in visual reconstruction (rFID 0.08) and achieves top-tier generation performance on GenEval (0.86), DPG-Bench (86.4), and WISE (0.53), while simultaneously outperforming previous native UMMs by an average of 10.0 points across eight challenging understanding benchmarks.

多模态生成理解视觉模型统一框架

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。