arXiv:2412.10702cs.CV2024-12AAAI被引 1

用自适应路由减少内存,让高分辨率图像抠图更快更省资源。

Memory Efficient Matting with Adaptive Token Routing

  • 通过路由器只让关键像素参与全局注意力,其余交由轻量模块处理。
  • 在Composition-1K上内存降低88%、延迟减少50%,精度仍领先。
  • 专为超高分辨率设计,适合需要高效抠图的工业级应用。

基于Transformer的模型在图像抠图中表现优异,但其全局自注意力的二次复杂度限制了在高分辨率图像上的应用。为此,我们提出MEMatte——一个面向高分辨率图像的内存高效抠图框架。MEMatte在每个全局注意力块前引入路由器,将信息丰富的令牌导向全局注意力,其余则路由至轻量令牌精修模块(LTRM)。路由器采用局部-全局策略预测每个令牌的路由概率,而LTRM使用高效模块模拟全局注意力。此外,我们提出批处理约束的自适应令牌路由(BATR)机制,使路由器能根据图像内容和网络阶段动态调整路由策略。我们还构建了超高清图像抠图数据集UHR-395,包含35,500张训练图与1,000张测试图,平均分辨率达4872×6017,由395种不同透明度图在11类背景上合成,均经高质量人工标注。大量实验表明,MEMatte在高分辨率及真实世界数据集上均优于现有方法,在Composition-1K上内存减少约88%、延迟降低50%。代码已开源:https://github.com/linyiheng123/MEMatte。

原文摘要 · Abstract (English)

Transformer-based models have recently achieved outstanding performance in image matting. However, their application to high-resolution images remains challenging due to the quadratic complexity of global self-attention. To address this issue, we propose MEMatte, a \textbf{m}emory-\textbf{e}fficient \textbf{m}atting framework for processing high-resolution images. MEMatte incorporates a router before each global attention block, directing informative tokens to the global attention while routing other tokens to a Lightweight Token Refinement Module (LTRM). Specifically, the router employs a local-global strategy to predict the routing probability of each token, and the LTRM utilizes efficient modules to simulate global attention. Additionally, we introduce a Batch-constrained Adaptive Token Routing (BATR) mechanism, which allows each router to dynamically route tokens based on image content and the stages of attention block in the network. Furthermore, we construct an ultra high-resolution image matting dataset, UHR-395, comprising 35,500 training images and 1,000 test images, with an average resolution of $4872\times6017$. This dataset is created by compositing 395 different alpha mattes across 11 categories onto various backgrounds, all with high-quality manual annotation. Extensive experiments demonstrate that MEMatte outperforms existing methods on both high-resolution and real-world datasets, significantly reducing memory usage by approximately 88% and latency by 50% on the Composition-1K benchmark. Our code is available at https://github.com/linyiheng123/MEMatte.

图像抠图Transformer内存优化高分辨率

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。