受大脑整合全局与局部信息启发，提出分步补丁合并方法提升视觉Transformer性能。
Brain-Inspired Stepwise Patch Merging for Vision Transformers
- 通过多尺度聚合与引导局部增强实现分步补丁合并
- 在ImageNet-1K、COCO等数据集上显著提升检测与分割精度
- 适用于各类骨干网络，尤其适合密集预测任务
层次化架构已成为视觉Transformer的主流设计范式，补丁合并（Patch Merging）是将柱状结构转变为层次化结构的关键组件。受大脑整合全局与局部信息以实现全面视觉理解能力的启发，我们提出分步补丁合并（Stepwise Patch Merging, SPM），增强后续注意力机制的感知能力。SPM由多尺度聚合（MSA）和引导局部增强（GLE）构成，有效平衡长程依赖建模与局部特征增强。在ImageNet-1K、COCO和ADE20K等基准数据集上的大量实验表明，SPM显著提升了多种模型的性能，尤其在目标检测和语义分割等密集预测任务中表现突出。同时，实验显示将SPM与不同骨干网络结合可进一步提升性能。代码已公开于https://github.com/Yonghao-Yu/StepwisePatchMerging。
原文摘要 · Abstract (English)
The hierarchical architecture has become a mainstream design paradigm for Vision Transformers (ViTs), with Patch Merging serving as the pivotal component that transforms a columnar architecture into a hierarchical one. Drawing inspiration from the brain's ability to integrate global and local information for comprehensive visual understanding, we propose Stepwise Patch Merging (SPM), which enhances the subsequent attention mechanism's ability to 'see' better. SPM consists of Multi-Scale Aggregation (MSA) and Guided Local Enhancement (GLE) striking a proper balance between long-range dependency modeling and local feature enhancement. Extensive experiments conducted on benchmark datasets, including ImageNet-1K, COCO, and ADE20K, demonstrate that SPM significantly improves the performance of various models, particularly in dense prediction tasks such as object detection and semantic segmentation. Meanwhile, experiments show that combining SPM with different backbones can further improve performance. The code has been released at https://github.com/Yonghao-Yu/StepwisePatchMerging.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。