用预训练特征空间的扩散模型,无需复杂设计即可高效生成图像。
RiT: Vanilla Diffusion Transformers Suffice in Representation Space

- 在冻结的DINOv2特征空间中直接做x-预测扩散建模。
- 图像生成质量超越同类方法,参数减少19%且推理更快。
- 适合追求高效生成与轻量化模型的研究者和开发者。
以x-预测为基础的流匹配方法(即回归干净数据点而非环境速度)被证明能有效利用像素空间中的低维流形结构。本文探讨预训练表示空间是否同样具备更优的流匹配学习分布。通过沿四个几何轴比较像素、SD-VAE与DINOv2特征,发现像素与DINOv2具有相近的内在维度(均≈33),但DINOv2在有效秩(7.3倍更高)、协方差条件数(35倍更优)、超额峰度(11.5倍更低)及流形内插误差(1.7倍更低)方面显著更优;而SD-VAE特征居中,表明优势源于表示学习目标而非单纯压缩。这些统计特性使流匹配回归更稳定,无需以往DINOv2扩散方法中的特殊预测头或黎曼传输。我们提出“表示图像变换器”(RiT):仅用x-预测训练的普通扩散变压器,基于冻结的DINOv2特征,并加入维度感知噪声调度与联合[CLS]-patch建模。在ImageNet 256×256上,RiT无引导时FID为1.45,有分类器自由引导时达1.14,参数量比DiT^DH-XL少19%(676M vs. 839M)。其对应的常微分方程可在粗离散化下高效求解:使用分类器自由引导时,5步Heun法即达FID 2.0,10步达1.25,无需蒸馏或一致性训练。代码见https://github.com/lezhang7/RiT。
原文摘要 · Abstract (English)
Flow matching with $x$-prediction -- regressing the clean data point rather than the ambient velocity -- is known to exploit low-dimensional manifold structure effectively in pixel space \cite{li2025back}. We ask whether a pretrained representation space, while containing a low-dimensional data manifold of comparable intrinsic dimensionality, offers a distribution more favorable for flow-matching learning. Comparing pixel, SD-VAE, and DINOv2 features along four geometric axes, we find that pixel and DINOv2 share nearly identical intrinsic dimensionalities (both $\hat{d}\!\approx\!33$) yet DINOv2 exhibits $7.3\times$ higher effective rank, $35\times$ better covariance conditioning, $11.5\times$ lower excess kurtosis, and $1.7\times$ lower on-manifold interpolation error; SD-VAE latents are consistently intermediate, indicating that the advantage stems from representation-learning objectives rather than mere compression. These statistical properties render the flow-matching regression well-conditioned and remove the need for the specialized prediction heads or Riemannian transport used by prior DINOv2 diffusion methods. We propose the \emph{Representation Image Transformer} (RiT): a vanilla Diffusion Transformer trained by $x$-prediction on frozen DINOv2 features, augmented only by a dimension-aware noise schedule and joint \texttt{[CLS]}-patch modeling. On ImageNet $256{\times}256$, RiT attains FID 1.45 without guidance and 1.14 with classifier-free guidance, outperforming DiT$^\text{DH}$-XL with $19\%$ fewer parameters (676M vs.\ 839M). The resulting ODE is efficiently solvable at coarse discretizations: with classifier-free guidance, $5$ Heun steps already reach FID 2.0 and $10$ steps reach 1.25, without distillation or consistency training. Code at https://github.com/lezhang7/RiT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。