arXiv:2501.15619cs.CVcs.AI2025-01被引 9

用2D高斯点图实现高效图像分块,提升视觉表示能力。

GaussianToken: An Effective Image Tokenizer with 2D Gaussian Splatting

  • 将图像编码为可调节位置、旋转、缩放的2D高斯点
  • 在高斯特征上进行量化并拼接参数,实现高质量重建
  • 适合需要精细视觉表征的多模态任务

有效的图像分块对多模态理解与生成任务至关重要,因需与离散文本数据对齐。现有方法使用向量量化(VQ)将像素映射到离散码本,并从离散表示重构图像,但受限于有限的离散码本空间,其表达能力较连续潜在空间弱。本文提出GaussianToken:一种基于2D高斯点图的高效图像分块方法。将编码样本表示为多个灵活的带特征2D高斯点,包含位置、旋转角、缩放因子和特征系数。对高斯特征采用标准量化,再将其量化结果与原始高斯参数拼接,经点图操作和后续解码模块完成重构。GaussianToken将2D高斯分布的局部影响融入离散空间,显著增强图像分块的表达能力。在CIFAR、Mini-ImageNet和ImageNet-1K上的重建表现达到竞争力水平。代码已开源:https://github.com/ChrisDong-THU/GaussianToken。

原文摘要 · Abstract (English)

Effective image tokenization is crucial for both multi-modal understanding and generation tasks due to the necessity of the alignment with discrete text data. To this end, existing approaches utilize vector quantization (VQ) to project pixels onto a discrete codebook and reconstruct images from the discrete representation. However, compared with the continuous latent space, the limited discrete codebook space significantly restrict the representational ability of these image tokenizers. In this paper, we propose GaussianToken: An Effective Image Tokenizer with 2D Gaussian Splatting as a solution. We first represent the encoded samples as multiple flexible featured 2D Gaussians characterized by positions, rotation angles, scaling factors, and feature coefficients. We adopt the standard quantization for the Gaussian features and then concatenate the quantization results with the other intrinsic Gaussian parameters before the corresponding splatting operation and the subsequent decoding module. In general, GaussianToken integrates the local influence of 2D Gaussian distribution into the discrete space and thus enhances the representation capability of the image tokenizer. Competitive reconstruction performances on CIFAR, Mini-ImageNet, and ImageNet-1K demonstrate the effectiveness of our framework. Our code is available at: https://github.com/ChrisDong-THU/GaussianToken.

图像分块2D高斯视觉表示

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。