提出分层图像标记法,实现单次前向传播的多尺度超分辨率。
Hierarchical Image Tokenization for Multi-Scale Image Super Resolution

- 分层标记机制逐级表示图像,跨尺度强制重叠以保留细节。
- 300万参数模型在无外部数据下达顶尖性能,支持多尺度输出。
- 仅需低分辨率与高分辨率配对数据,适合资源受限场景。
我们提出一种基于视觉自回归(VAR)建模的多尺度图像超分辨率(ISR)方法。传统方法利用残差量化(RQ)逐级生成图像,但受限于固定尺度输出,且依赖大型骨干网络或大量标注数据。为此,本文引入两项新组件:一是分层图像标记(HIT),通过跨尺度强制标记重叠,逐步表示不同分辨率下的图像;二是直接偏好优化(DPO)正则项,仅基于(低分辨率,高分辨率)成对数据,引导Transformer优先生成高分辨率结果。HIT为训练提供强先验,使模型仅需300万参数即可达到10亿参数模型(VARSR)的性能,在无需外部数据情况下实现单次前向传播的多尺度输出,显著提升灵活性与效率。
原文摘要 · Abstract (English)
We introduce a multi-scale Image Super Resolution (ISR) method building on recent advances in Visual Auto-Regressive (VAR) modeling. VAR models break image tokenization into additive, gradually increasing scales, using Residual Quantization (RQ), an approach that aligns perfectly with our target ISR task. Previous works taking advantage of this synergy suffer from two main shortcomings. First, due to the limitations in RQ, they only generate images at a predefined fixed scale, failing to map intermediate outputs to the corresponding image scales. They also rely on large backbones or a large corpus of annotated data to achieve better performance. To address both shortcomings, we introduce two novel components to the VAR training for ISR, aiming at increasing its flexibility and reducing its complexity. In particular, we introduce a) a \textbf{Hierarchical Image Tokenization (HIT)} approach that progressively represents images at different scales while enforcing token overlap across scales, and b) a \textbf{Direct Preference Optimization (DPO) regularization term} that, relying solely on the (LR,HR) pair, encourages the transformer to produce the latter over the former. Our proposed HIT acts as a strong inductive bias for the VAR training, resulting in a small model (300M params vs 1B params of VARSR), that achieves state-of-the-art results without external training data, and that delivers multi-scale outputs with a single forward pass.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。