用可学习的高斯分布实现图像区域自适应分块,提升生成质量与效率。
GPSToken: Gaussian Parameterized Spatially-adaptive Tokenization for Image Representation and Generation
- 用2D高斯参数动态建模图像区域的位置、形状和纹理。
- 128个令牌在重建与生成任务中分别达到0.65和1.50的FID得分。
- 适合需要高效结构生成与纹理细化的图像生成场景。
有效的图像表示与生成依赖高效的分块策略。传统方法受限于均匀的2D/1D网格分块,难以灵活表征不同形状、纹理及位置的图像区域,影响特征表达能力。本文提出一种新型的高斯参数化空间自适应分块框架GPSToken,通过可学习的2D高斯函数动态建模图像区域的位置(均值)、形状(协方差)与纹理特征。首先采用熵驱动算法将图像划分为大小不一、纹理一致的区域;随后,每个区域由2D高斯参数化并结合纹理特征表示。专用Transformer优化高斯参数,实现位置/形状的连续调整与内容感知特征提取。解码阶段,通过可微分的splatting渲染器将高斯分块重构为2D特征图,实现与标准解码器的端到端训练。该方法将空间布局(高斯参数)与纹理特征解耦,支持两阶段生成:先用轻量网络合成结构布局,再基于结构生成纹理。实验表明,使用128个令牌,GPSToken在图像重建与生成任务中分别取得rFID 0.65与FID 1.50的当前最优表现。
原文摘要 · Abstract (English)
Effective and efficient tokenization plays an important role in image representation and generation. Conventional methods, constrained by uniform 2D/1D grid tokenization, are inflexible to represent regions with varying shapes and textures and at different locations, limiting their efficacy of feature representation. In this work, we propose $\textbf{GPSToken}$, a novel $\textbf{G}$aussian $\textbf{P}$arameterized $\textbf{S}$patially-adaptive $\textbf{Token}$ization framework, to achieve non-uniform image tokenization by leveraging parametric 2D Gaussians to dynamically model the shape, position, and textures of different image regions. We first employ an entropy-driven algorithm to partition the image into texture-homogeneous regions of variable sizes. Then, we parameterize each region as a 2D Gaussian (mean for position, covariance for shape) coupled with texture features. A specialized transformer is trained to optimize the Gaussian parameters, enabling continuous adaptation of position/shape and content-aware feature extraction. During decoding, Gaussian parameterized tokens are reconstructed into 2D feature maps through a differentiable splatting-based renderer, bridging our adaptive tokenization with standard decoders for end-to-end training. GPSToken disentangles spatial layout (Gaussian parameters) from texture features to enable efficient two-stage generation: structural layout synthesis using lightweight networks, followed by structure-conditioned texture generation. Experiments demonstrate the state-of-the-art performance of GPSToken, which achieves rFID and FID scores of 0.65 and 1.50 on image reconstruction and generation tasks using 128 tokens, respectively. Codes and models of GPSToken can be found at $\href{https://github.com/xtudbxk/GPSToken}{https://github.com/xtudbxk/GPSToken}$.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。