arXiv:2605.02794cs.CV2026-05

用轻量态空间模型替代部分注意力层,让图像修复更快更省电。

Edge-Efficient Image Restoration: Transformer Distillation into State-Space Models

论文配图:Edge-Efficient Image Restoration: Transformer Distillation into State-Space Models
图 1 · 摘自论文原文
  • 用状态空间模型蒸馏变压器块,构建混合修复架构
  • 在高通骁龙8上提速1.17到3.4倍,精度仍在线
  • 自动搜索最优组合,无需反复测硬件延迟

我们提出一种模块化混合图像修复框架,融合变压器与状态空间模型(SSM)块,重点提升边缘设备上的运行效率。尽管变压器通过自注意力机制具备强全局建模能力,但其注意力核在移动设备上带来显著延迟,尤其对高分辨率输入。相比之下,如Mamba等状态空间模型可实现线性时间序列建模,运行开销更低,但在细粒度修复任务中表现可能不足。为平衡精度与效率,我们训练轻量级的SSM块作为变压器块的特征蒸馏代理,并用于构建类U-Net的混合架构。为自动发现有效块组合,引入高效网络搜索(ENS),一种多目标搜索策略,从预对齐组件中选取特定任务的混合配置。ENS在优化修复质量的同时惩罚变压器使用,作为延迟的轻量级代理,实现无需重复硬件性能测试的架构发现。在高通骁龙8精英版CPU上,Restormer基线推理耗时10119.52毫秒。相比之下,由ENS发现的混合模型显著降低运行时间:ENS-去模糊仅需2973毫秒(快3.4倍),ENS-去雨为5816毫秒(快1.74倍),ENS-去噪为8666毫秒(快1.17倍),同时保持竞争力的修复质量。

原文摘要 · Abstract (English)

We propose a modular framework for hybrid image restoration that integrates transformer and state-space model (SSM) blocks with a focus on improving runtime efficiency on edge hardware. While transformers provide strong global modeling through self-attention, their attention kernels incur substantial latency on mobile devices, especially for high-resolution inputs. In contrast, SSMs such as Mamba offer lineartime sequence modeling with lower runtime overhead but may underperform on fine grained restoration tasks. To balance accuracy and efficiency, we train lightweight SSM blocks as feature-distilled surrogates of transformer blocks and use them to construct hybrid U-Net-style architectures. To automatically discover effective block combinations, we introduce Efficient Network Search (ENS), a multi-objective search strategy that selects task-specific hybrid configurations from pre-aligned components. ENS optimizes restoration quality while penalizing transformer usage, serving as a lightweight proxy for latency and enabling architecture discovery without repeated hardware profiling. On a Snapdragon 8 Elite CPU, the Restormer baseline requires 10119.52 ms for inference. In contrast, ENS-discovered hybrids significantly reduce runtime: ENS-Deblurring runs in 2973 ms (3.4x faster), ENS-Deraining in 5816 ms (1.74x faster), and ENS-Denoising in 8666 ms (1.17x faster), while maintaining competitive restoration quality.

图像修复边缘计算模型压缩状态空间

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。