arXiv:2411.10251cs.CV2024-11

提出MAGA机制,让ViT更好保留头发等细结构

Morpho-Aware Global Attention for Image Matting

  • 用类俄罗斯方块卷积对齐细结构形状,保持局部对应
  • 在全局注意力中强调局部形态,使细节更清晰
  • 适合需要高精度抠图的应用,如影视特效

视觉变换器(ViTs)和卷积神经网络(CNNs)在图像抠图中面临固有挑战,尤其难以保持精细结构细节。ViTs虽具全局感受野,但常丢失发丝等局部细节;而CNN受限于局部感受野,需深层网络近似全局上下文,却在深层难以保留细结构。为此,本文提出一种新型形态感知全局注意力(MAGA)机制,通过类俄罗斯方块的卷积模式对齐细结构局部形状,确保最优局部对应并保持对形态细节的敏感性。提取的局部形态信息作为查询嵌入,投影到全局键嵌入上,强化在全局上下文中对局部细节的关注;再投影到值嵌入,将强调的形态细节无缝融合至统一全局结构中。该方法可同时关注局部形态并整合为连贯整体,有效保留精细结构。大量实验表明,基于MAGA的ViT在两个基准上均显著优于现有方法,平均提升4.3%(SAD)和39.5%(MSE)。

原文摘要 · Abstract (English)

Vision Transformers (ViTs) and Convolutional Neural Networks (CNNs) face inherent challenges in image matting, particularly in preserving fine structural details. ViTs, with their global receptive field enabled by the self-attention mechanism, often lose local details such as hair strands. Conversely, CNNs, constrained by their local receptive field, rely on deeper layers to approximate global context but struggle to retain fine structures at greater depths. To overcome these limitations, we propose a novel Morpho-Aware Global Attention (MAGA) mechanism, designed to effectively capture the morphology of fine structures. MAGA employs Tetris-like convolutional patterns to align the local shapes of fine structures, ensuring optimal local correspondence while maintaining sensitivity to morphological details. The extracted local morphology information is used as query embeddings, which are projected onto global key embeddings to emphasize local details in a broader context. Subsequently, by projecting onto value embeddings, MAGA seamlessly integrates these emphasized morphological details into a unified global structure. This approach enables MAGA to simultaneously focus on local morphology and unify these details into a coherent whole, effectively preserving fine structures. Extensive experiments show that our MAGA-based ViT achieves significant performance gains, outperforming state-of-the-art methods across two benchmarks with average improvements of 4.3% in SAD and 39.5% in MSE.

图像抠图视觉变换器细结构保留注意力机制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。