arXiv:2608.00626cs.CV2026-08

不同图像生成目标表示的难易分布差异显著,影响训练与推理效率。

Where Does Generative Difficulty Reside? An Empirical Study of Target Representations

论文配图:Where Does Generative Difficulty Reside? An Empirical Study of Target Representations
图 1 · 摘自论文原文
  • 统一模型下对比像素、VAE、DINOv2等表示空间的生成效果
  • DINOv2收敛最快但需更强局部去噪与上下文融合机制
  • 生成难度由上下文建模、去噪和分布控制共同决定,非单一因素

目标表示定义了图像生成器需学习的分布,但常被视为可互换接口。这一假设对连续掩码生成器尤其存疑,因其结合可见令牌的上下文推断与缺失令牌的条件建模。我们在统一的掩码自回归修正流模型中,研究了原始像素、SD-VAE隐变量、DINOv2特征以及MAE表示-自编码器特征。在相同的ImageNet训练预算下,这些表示空间表现出不同的优化与推理规律:DINOv2在迭代次数与计算量上收敛最快,但对更宽的局部去噪器和直接上下文融合有强依赖;像素表示优化显著更慢,且需不同预测、掩码与引导配置;MAE重建更忠实并呈现明显语义聚类,但生成质量远低于DINOv2。各表示对无分类器引导的响应不同,占据不同的精度-召回权衡。结果表明,压缩率、重建保真度、令牌维度与可见语义聚类均不能单独预测生成行为。相反,目标表示将生成难度重新分配至上下文建模、单令牌去噪与推理时分布控制之间。

原文摘要 · Abstract (English)

The target representation defines the distribution an image generator must learn, yet it is often treated as an interchangeable interface. This assumption is particularly questionable for continuous masked generators, which combine contextual inference from visible tokens with conditional modeling of each missing token. We study raw pixels, SD-VAE latents and DINOv2 as well as MAE representation-autoencoder features within a unified masked autoregressive rectified-flow model. Under a shared ImageNet training budget, these spaces exhibit distinct optimization and inference regimes. DINOv2 converges fastest in both iterations and computation but benefits strongly from a wider local denoiser and direct context fusion. Pixels optimize substantially more slowly and require a different prediction, masking, and guidance configuration. MAE reconstructs images more faithfully and exhibits clear semantic clustering, yet produces generations substantially worse than DINOv2. The representations also respond differently to classifier-free guidance and occupy distinct precision-recall trade-offs. Together, our results show that compression, reconstruction fidelity, token dimensionality, and visible semantic clustering do not individually predict generative behavior. Instead, target representations redistribute difficulty across contextual modeling, per-token denoising, and inference-time distributional control.

生成模型表示学习扩散模型图像生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。