arXiv:2605.10780cs.CVcs.AI2026-05

融合多层视觉特征,提升图像重建与生成质量

Beyond the Last Layer: Multi-Layer Representation Fusion for Visual Tokenization

论文配图:Beyond the Last Layer: Multi-Layer Representation Fusion for Visual Tokenization
图 1 · 摘自论文原文
  • 通过自适应路由融合编码器所有层级特征,恢复丢失的细节信息
  • 在ImageNet-256上将rFID从0.57降至0.29,生成FID提升至1.65
  • 揭示特征丰富度与重建质量的可预测线性关系,适合图像生成研究者

利用冻结的预训练视觉编码器作为视觉分词器的表示自编码器已取得优异的重建与生成效果。然而,现有方法仅提取最后一层特征,忽略了中间层分布的丰富层次信息。我们发现,低级视觉细节在最后一层仅以衰减残差形式存在,通过显式融合多层特征可显著恢复这些信息。为此提出轻量级融合模块DRoRAE,通过能量约束路由和增量修正,自适应聚合所有编码器层,生成适配冻结解码器的增强潜在表示。采用三阶段解耦训练策略:先在冻结解码器隐式分布约束下学习融合,再微调解码器以充分挖掘增强表示。在ImageNet-256上,rFID由0.57降至0.29,生成FID从1.74提升至1.65(含AutoGuidance),增益亦延伸至文生图合成。此外,我们发现融合容量与重建质量间存在对数线性规律(R²=0.86),揭示‘表示丰富度’是类比自然语言处理中词表大小的新可预测扩展维度。

原文摘要 · Abstract (English)

Representation autoencoders that reuse frozen pretrained vision encoders as visual tokenizers have achieved strong reconstruction and generation quality. However, existing methods universally extract features from only the last encoder layer, discarding the rich hierarchical information distributed across intermediate layers. We show that low-level visual details survive in the last layer merely as attenuated residuals after multiple layers of semantic abstraction, and that explicitly fusing multi-layer features can substantially recover this lost information. We propose DRoRAE (Depth-Routed Representation AutoEncoder), a lightweight fusion module that adaptively aggregates all encoder layers via energy-constrained routing and incremental correction, producing an enriched latent compatible with a frozen pretrained decoder. A three-phase decoupled training strategy first learns the fusion under the implicit distributional constraint of the frozen decoder, then fine-tunes the decoder to fully exploit the enriched representation. On ImageNet-256, DRoRAE reduces rFID from 0.57 to 0.29 and improves generation FID from 1.74 to 1.65 (with AutoGuidance), with gains also transferring to text-to-image synthesis. Furthermore, we uncover a log-linear scaling law ($R^2{=}0.86$) between fusion capacity and reconstruction quality, identifying \textit{representation richness} as a new, predictably scalable dimension for visual tokenizers analogous to vocabulary size in NLP.

视觉分词多层融合自编码器图像生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。