无需标注框的多查询场景文字检索方法,提升跨类型查询兼容性
MSTAR: Box-free Multi-query Scene Text Retrieval with Attention Recycling
- 采用渐进视觉嵌入与注意力复用,动态捕捉多粒度文本表征
- 在7个公开数据集上超越前序模型,总文本数据集上提升6.4%的MAP
- 首个支持四类查询的多查询文字检索基准,适合多场景应用研究
场景文字检索得益于精确的文本定位取得了显著进展,但现有方法通常需要昂贵的边界框标注进行训练。此外,多数方法采用定制化检索策略,难以统一多种查询类型以满足多样化需求。为此,我们提出无框多查询场景文字检索方法(MSTAR),通过渐进式视觉嵌入动态捕捉文本的多粒度表征,并融合自由风格文本查询与风格感知指令。同时,引入多实例匹配模块增强视觉-语言对齐能力。此外,我们构建了首个评估多查询场景文字检索能力的基准数据集MQTR,包含四种查询类型和16,000张图像。大量实验表明,该方法在七个公共数据集及MQTR数据集上均表现优异。值得注意的是,MSTAR在Total-Text数据集上以6.4%的绝对提升超越此前最先进模型,且无需边界框标注;在MQTR基准上平均性能领先8.5%。代码与数据集已开源。
原文摘要 · Abstract (English)
Scene text retrieval has made significant progress with the assistance of accurate text localization. However, existing approaches typically require costly bounding box annotations for training. Besides, they mostly adopt a customized retrieval strategy but struggle to unify various types of queries to meet diverse retrieval needs. To address these issues, we introduce Muti-query Scene Text retrieval with Attention Recycling (MSTAR), a box-free approach for scene text retrieval. It incorporates progressive vision embedding to dynamically capture the multi-grained representation of texts and harmonizes free-style text queries with style-aware instructions. Additionally, a multi-instance matching module is integrated to enhance vision-language alignment. Furthermore, we build the Multi-Query Text Retrieval (MQTR) dataset, the first benchmark designed to evaluate the multi-query scene text retrieval capability of models, comprising four query types and 16k images. Extensive experiments demonstrate the superiority of our method across seven public datasets and the MQTR dataset. Notably, MSTAR marginally surpasses the previous state-of-the-art model by 6.4% in MAP on Total-Text while eliminating box annotation costs. Moreover, on the MQTR benchmark, MSTAR significantly outperforms the previous models by an average of 8.5%. The code and datasets are available at https://github.com/yingift/MSTAR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。