arXiv:2608.06832cs.CV2026-08中稿 · ICML

提出可感知退化分布的变形令牌化方法,提升图像修复模型对复杂退化的适应能力。

Bend the Basics: Degradation-Aware Deformable Tokenization for All-in-One Image Restoration

论文配图:Bend the Basics: Degradation-Aware Deformable Tokenization for All-in-One Image Restoration
图 1 · 摘自论文原文
  • 通过轻量退化编码器生成全局退化向量与空间退化图,动态调节令牌化与重构过程。
  • 在五项基准上平均PSNR达30.72 dB(五退化场景),优于现有方法0.5~1.1 dB。
  • 支持退化自适应可视化,适合需要精准退化建模的图像修复任务。

全功能图像修复旨在用单一模型恢复多种、空间非均匀退化的图像。然而,许多统一的Transformer依赖固定块划分:任务/退化条件仅在令牌化后注入主干模块,导致嵌入与重建阶段对局部退化变化不敏感。相比之下,本文提出柔性图像Transformer(FIT),在整个流程中显式建模退化感知能力,从块采样到像素重构均实现退化自适应。具体而言,FIT利用轻量退化编码器从局部退化强度预测全局退化向量 $oldsymbol{g}$ 和空间退化图 $oldsymbol{M}$,联合调控嵌入与反嵌入过程中的自适应形变。为增强跨退化类型的鲁棒性,引入任务令牌丢弃策略,在训练中正则化任务条件。在五个标准基准(BSD68, Rain100L, SOTS, GoPro, LOLv1)上,FIT在五退化设置下达到30.72 dB平均PSNR,三退化设置下达32.83 dB,较近期统一修复方法提升0.5~1.1 dB。此外,学习到的偏移量可直接用于可视化退化感知的空间适应性。

原文摘要 · Abstract (English)

All-in-one image restoration seeks a single model that can recover images degraded by diverse and spatially non-uniform corruptions. However, many unified Transformers rely on fixed patch partitioning: task/degradation condition is injected only into the backbone blocks after tokenization, leaving the embedding and reconstruction stages insensitive to local degradation variations. In contrast to previous approaches, we present Flexible Image Transformer (FIT) that explicitly models degradation awareness across the entire pipeline, from patch sampling to pixel reconstruction. Specifically, FIT employs a lightweight Degradation Encoder to predict a global degradation vector $\mathbf{g}$ and a spatial degradation map $\mathbf{M}$ from local degradation severity, which jointly condition the patch embedding and unembedding through adaptive deformation. Moreover, to improve robustness across degradation types, we introduce a task-token dropout strategy that regularizes task conditioning during training. On five standard benchmarks (BSD68, Rain100L, SOTS, GoPro, and LOLv1), FIT achieves state-of-the-art performance with 30.72 dB average PSNR on the five-degradation setting and 32.83 dB on the three-degradation setting, outperforming recent unified restoration methods by +0.5$\sim$1.1 dB. Moreover, the learned offsets provide a direct handle for visualizing degradation-aware spatial adaptation.

图像修复退化感知变形令牌化Transformer

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。