arXiv:2506.04990cs.CV2025-06被引 1

用单模型实现多尺度图像超分,效果更优且无需额外数据

Multi-scale Image Super Resolution with a Single Auto-Regressive Model

  • 设计分层图像标记化,让模型跨尺度保持语义一致
  • 引入偏好优化训练,仅用高低分辨率标记即实现超分
  • 300万参数小模型达成顶尖性能,支持半倍与全倍放大

本文研究图像超分辨率(ISR),利用视觉自回归(VAR)建模的最新进展。传统方法中,VAR通过渐进式预测下一尺度残差来工作,预训练中学习到的强先验与下游任务高度契合。现有方案VARSR虽表现良好,但受限于残差量化器,仅能固定分辨率处理,无法映射中间输出至对应图像尺度;且依赖10亿参数大模型(VAR-d24)及大规模私有数据集。为此,本文提出两项新组件:一是分层图像标记化方法,采用多尺度标记器逐步表示不同尺度图像,并强制跨尺度标记重叠;二是直接偏好优化(DPO)正则项,仅基于低/高分辨率标记化结果,引导模型生成高质量输出。据我们所知,这是首次训练量化器以强制跨尺度语义一致残差,也是首次将偏好优化用于训练自回归模型。结合二者,模型可在一次前向传播中完成去噪与半倍、全倍超分。在不使用外部训练数据的情况下,仅用300万参数即达到当前最优效果。

原文摘要 · Abstract (English)

In this paper we tackle Image Super Resolution (ISR), using recent advances in Visual Auto-Regressive (VAR) modeling. VAR iteratively estimates the residual in latent space between gradually increasing image scales, a process referred to as next-scale prediction. Thus, the strong priors learned during pre-training align well with the downstream task (ISR). To our knowledge, only VARSR has exploited this synergy so far, showing promising results. However, due to the limitations of existing residual quantizers, VARSR works only at a fixed resolution, i.e. it fails to map intermediate outputs to the corresponding image scales. Additionally, it relies on a 1B transformer architecture (VAR-d24), and leverages a large-scale private dataset to achieve state-of-the-art results. We address these limitations through two novel components: a) a Hierarchical Image Tokenization approach with a multi-scale image tokenizer that progressively represents images at different scales while simultaneously enforcing token overlap across scales, and b) a Direct Preference Optimization (DPO) regularization term that, relying solely on the LR and HR tokenizations, encourages the transformer to produce the latter over the former. To the best of our knowledge, this is the first time a quantizer is trained to force semantically consistent residuals at different scales, and the first time that preference-based optimization is used to train a VAR. Using these two components, our model can denoise the LR image and super-resolve at half and full target upscale factors in a single forward pass. Additionally, we achieve \textit{state-of-the-art results on ISR}, while using a small model (300M params vs ~1B params of VARSR), and without using external training data.

图像超分自回归模型多尺度小模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。