首个统一图像视频的视觉分词器,提升多模态模型理解生成能力
HYDRA-X: Native Unified Multimodal Models with Holistic Visual Tokenizers

- 设计统一视觉分词器,将图像视频输入融合到单一ViT中
- 帧级因果时序注意力+分层时序压缩,重建性能更优
- 在7B模型上实现跨图像视频任务强性能,适合多模态研究者
统一多模态模型(UMMs)的核心在于全局视觉分词器,它将多样化的视觉输入映射到统一表示空间。本文提出HYDRA-X,首个在单个视觉变换器(ViT)中统一图像与视频分词的UMM。针对两大挑战:高效注入时空重建能力,以及在隐空间嵌入图像与视频级语义感知。实验表明:(1) 帧级因果时序注意力足以支持视觉重建,全时空注意力反而降低性能;(2) 分层时序压缩显著优于单步方案。为解决第二点,提出轻量级解压器,在图像-视频联合教师监督下上采样时序压缩特征,强化紧凑隐空间中的互补语义结构。基于此分词器,进一步提出改进编辑流程:源目标交互应在分词器的隐空间内进行,而非语言模型的语义层,显著提升编辑一致性并加速收敛。在7B密集模型上,HYDRA-X在图像与视频理解及生成任务中表现优异,为未来统一分词器的UMM奠定基础。
原文摘要 · Abstract (English)
Holistic visual tokenizers are fundamental to unified multimodal models (UMMs) as they map diverse visual inputs into a unified representation space. In this paper, we present HYDRA-X, the first UMM that unifies image and video tokenization within a single Vision Transformer (ViT). Our design is driven by two core challenges: efficiently injecting spatiotemporal reconstruction capability into a native ViT, and embedding image- and video-level semantic awareness into the latent space. To address the first, comprehensive ablations reveal two key findings: (1) frame-level causal temporal attention suffices for visual reconstruction, whereas full spatiotemporal attention degrades it; and (2) hierarchical temporal compression substantially outperforms single-step alternatives. To tackle the second, we propose a lightweight decompressor that upsamples temporally compressed features under joint image-video teacher supervision, thereby enforcing complementary semantic structures within the compact latent space. Building on this holistic tokenizer, we further propose a principled improvement of the editing pipeline: source-target interaction should occur at the latent level inside the tokenizer rather than at the semantic level inside the LLM, substantially improving editing consistency and accelerating convergence. Instantiated at the 7B dense model, HYDRA-X achieves strong performance across image and video understanding and generation tasks, paving the way for future unified-tokenizer UMMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。