用2D高斯模型提升图像令牌化,更好保留几何结构。
2D Gaussians Meet Visual Tokenizer
- 将2D高斯分布融入量化编码框架,显式建模位置、旋转和尺度。
- 在ImageNet 256x256上实现rFID 0.556和PSNR 24.93的顶尖重建效果。
- 适合关注图像结构建模与高效视觉表示的研究者。
图像令牌化是AR图像生成的关键组件,决定视觉内容如何被编码为紧凑表示。现有基于量化的方法如VQ-GAN主要关注纹理和颜色等外观特征,因采用块状设计而常忽略几何结构。本文提出一种新范式——视觉高斯量化(VGQ),通过将2D高斯融入传统代码本量化框架,显式增强结构建模能力。该方法克服了传统量化方法因块状设计导致的结构建模不足问题,将图像潜在表示为2D高斯分布,直接建模位置、旋转和尺度等结构参数。实验表明,增加令牌内2D高斯密度可显著提升重建保真度,在ImageNet 256x256基准上达到rFID 0.556和PSNR 24.93,显著优于现有方法。
原文摘要 · Abstract (English)
The image tokenizer is a critical component in AR image generation, as it determines how rich and structured visual content is encoded into compact representations. Existing quantization-based tokenizers such as VQ-GAN primarily focus on appearance features like texture and color, often neglecting geometric structures due to their patch-based design. In this work, we explored how to incorporate more visual information into the tokenizer and proposed a new framework named Visual Gaussian Quantization (VGQ), a novel tokenizer paradigm that explicitly enhances structural modeling by integrating 2D Gaussians into traditional visual codebook quantization frameworks. Our approach addresses the inherent limitations of naive quantization methods such as VQ-GAN, which struggle to model structured visual information due to their patch-based design and emphasis on texture and color. In contrast, VGQ encodes image latents as 2D Gaussian distributions, effectively capturing geometric and spatial structures by directly modeling structure-related parameters such as position, rotation and scale. We further demonstrate that increasing the density of 2D Gaussians within the tokens leads to significant gains in reconstruction fidelity, providing a flexible trade-off between token efficiency and visual richness. On the ImageNet 256x256 benchmark, VGQ achieves strong reconstruction quality with an rFID score of 1.00. Furthermore, by increasing the density of 2D Gaussians within the tokens, VGQ gains a significant boost in reconstruction capability and achieves a state-of-the-art reconstruction rFID score of 0.556 and a PSNR of 24.93, substantially outperforming existing methods. Codes will be released soon.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。