统一框架用提示词实现图像校正与矫正,效果更优且通用性强。
Rectification Reimagined: A Unified Mamba Model for Image Correction and Rectangling with Prompts
- 基于统一畸变模型,用双组件结构处理几何变形和图像退化。
- 在多个数据集上超越现有方法,尤其在复杂畸变下表现更佳。
- 适合需要多任务图像修复的手机摄影系统开发者使用。
图像校正与矫正对智能手机等实际摄影系统至关重要。尽管深度学习带来显著性能提升,但现有方法多依赖特定任务架构,泛化能力受限。本文提出统一校正框架 UniRect,从统一畸变校正视角解决多项实际任务。通过模拟不同镜头类型,将多种任务特异性逆问题纳入通用畸变模型。UniRect采用无任务依赖的双组件结构:变形模块使用新型残差渐进薄板样条(RP-TPS)模型处理复杂几何变形;后续恢复模块则利用残差 Mamba 块(RMBs)抵消变形带来的退化,提升输出图像保真度。此外,设计稀疏专家混合(SMoEs)结构以缓解多任务学习中因畸变差异导致的任务竞争问题。大量实验表明,本模型在多个基准上达到当前最优性能。
原文摘要 · Abstract (English)
Image correction and rectangling are valuable tasks in practical photography systems such as smartphones. Recent remarkable advancements in deep learning have undeniably brought about substantial performance improvements in these fields. Nevertheless, existing methods mainly rely on task-specific architectures. This significantly restricts their generalization ability and effective application across a wide range of different tasks. In this paper, we introduce the Unified Rectification Framework (UniRect), a comprehensive approach that addresses these practical tasks from a consistent distortion rectification perspective. Our approach incorporates various task-specific inverse problems into a general distortion model by simulating different types of lenses. To handle diverse distortions, UniRect adopts one task-agnostic rectification framework with a dual-component structure: a {Deformation Module}, which utilizes a novel Residual Progressive Thin-Plate Spline (RP-TPS) model to address complex geometric deformations, and a subsequent Restoration Module, which employs Residual Mamba Blocks (RMBs) to counteract the degradation caused by the deformation process and enhance the fidelity of the output image. Moreover, a Sparse Mixture-of-Experts (SMoEs) structure is designed to circumvent heavy task competition in multi-task learning due to varying distortions. Extensive experiments demonstrate that our models have achieved state-of-the-art performance compared with other up-to-date methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。