arXiv:2607.06553cs.CV2026-07

用文本生成模型做密集预测,不转成图像,直接输出像素级结果。

From RGB Generation to Dense Field Readout: Pixel-Space Dense Prediction with Text-to-Image Models

论文配图:From RGB Generation to Dense Field Readout: Pixel-Space Dense Prediction with Text-to-Image Models
图 1 · 摘自论文原文
  • 将文本生成模型的token映射到像素块,直接输出任务原生数据
  • 在6个任务上达到新纪录,40亿参数下速度提升2.48倍
  • 适合需要高精度像素级输出的研究者和应用

大规模文本到图像模型因其在RGB生成预训练中学习到丰富的语义、结构与几何先验,成为密集预测的理想骨干。现有方法将密集预测视为目标生成:将深度、法向、透明度图等标注编码进已训练的VAE隐空间并解码为类似图像的目标。我们指出,这继承了生成式输出接口,而非密集预测所需——后者要求像素精确、任务原生的场量输出,而非新渲染内容。关键观察是,预训练的DiT已在图像平面上通过块-标记-块结构组织输入,每个标记对应固定输出块,其通道可承载任务原生量而非RGB外观。我们提出ReChannel:保留VAE编码器以匹配输入分布,移除目标侧解码器,用任务专用LoRA适配冻结的DiT,通过共享的局部线性头将每个标记映射至p×p×K_t像素块——约33K参数,无空间混合。使用FLUX-Klein,在六项密集预测任务和十余个基准上评估。此极简接口在无裁剪掩膜的抠图、KITTI深度估计和指代分割上达到新最优,对法向、显著性、姿态等任务保持竞争力。在匹配的40亿参数设置下,精度更高且快2.48倍。密集感知可受益于生成式预训练,而不必继承其输出接口。

原文摘要 · Abstract (English)

Large-scale text-to-image models are attractive backbones for dense prediction because RGB generation pretraining learns rich semantic, structural, and geometric priors. Existing generative and editing approaches reuse these priors by casting dense prediction as target generation: annotations such as depth, normals, alpha mattes, masks, and heatmaps are encoded into an RGB-trained VAE latent space and decoded back as image-like targets. We argue this inherits more of the generative output interface than dense prediction requires: unlike RGB synthesis, dense prediction asks for pixel-correct, task-native fields on the same image plane, not new RGB content to be rendered. Our key observation is that a pretrained DiT already organizes RGB inputs through a patch-to-token-to-patch lattice on the image plane, so each token indexes a fixed output patch whose channels can carry task-native quantities instead of RGB appearance. We instantiate this as ReChannel: we keep the VAE encoder for the DiT's input distribution but drop the target-side decoder, adapt the frozen DiT with task LoRA, and map each token to its p x p x K_t pixel-space patch through a shared token-local linear head--about 33K parameters, no spatial mixing. Using FLUX-Klein, we evaluate on six dense prediction tasks and over a dozen benchmarks. This minimal interface sets new state-of-the-art on trimap-free matting, KITTI depth, and referring segmentation, and stays competitive on normals, saliency, and pose. In a matched 4B setting it is more accurate and 2.48x faster than an edit-plus-latent-decode counterpart--dense perception can benefit from generative pretraining without inheriting its output interface.

密集预测文本生成扩散模型像素级输出

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。