通过区域文本提示增强细粒度文字检测,提升模型对小文字的识别能力。
Region Prompt Tuning: Fine-grained Scene Text Detection Utilizing Region Text Prompt
- 将文本拆分为字符,与视觉区域一一对应,聚焦局部细节
- 在ICDAR2015等数据集上达到领先性能,显著提升小文字检测率
- 适合需要高精度文字检测的场景,如OCR、自动驾驶
近期提示调优方法成功将CLIP等大规模模型应用于场景文字检测等下游任务。传统文本提示仅关注全局特征,忽略细粒度信息,导致小文字被遗漏。本文提出区域提示调优(RPT)方法,将区域文本提示分解为单个字符,并将视觉特征图划分为区域视觉标记,建立字符与标记的一一对应关系,使每个字符匹配对应区域的局部特征,避免细节丢失。为此,引入共享位置嵌入以关联字符与标记,并采用双向距离损失对齐提示字符与目标文本。在编码前与编码后均设计字符-标记交互模块,以细化细粒度信息。最终融合图像-文本生成的通用得分图与字符-标记匹配生成的区域得分图,形成兼顾全局与局部特征的最终得分图,并输入DBNet进行文字检测。在ICDAR2015、TotalText、CTW1500等基准数据集上的实验表明,RPT表现优异,验证了其在场景文字检测中的有效性。
原文摘要 · Abstract (English)
Recent advancements in prompt tuning have successfully adapted large-scale models like Contrastive Language-Image Pre-trained (CLIP) for downstream tasks such as scene text detection. Typically, text prompt complements the text encoder's input, focusing on global features while neglecting fine-grained details, leading to fine-grained text being ignored in task of scene text detection. In this paper, we propose the region prompt tuning (RPT) method for fine-grained scene text detection, where region text prompt proposed would help focus on fine-grained features. Region prompt tuning method decomposes region text prompt into individual characters and splits visual feature map into region visual tokens, creating a one-to-one correspondence between characters and tokens. This allows a character matches the local features of a token, thereby avoiding the omission of detailed features and fine-grained text. To achieve this, we introduce a sharing position embedding to link each character with its corresponding token and employ a bidirectional distance loss to align each region text prompt character with the target ``text''. To refine the information at fine-grained level, we implement character-token level interactions before and after encoding. Our proposed method combines a general score map from the image-text process with a region score map derived from character-token matching, producing a final score map that could balance the global and local features and be fed into DBNet to detect the text. Experiments on benchmarks like ICDAR2015, TotalText, and CTW1500 demonstrate RPT impressive performance, underscoring its effectiveness for scene text detection.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。