用联合训练提升文字定位与识别,特别适合艺术字体场景。
SupGRPO: Enhancing GRPO with Matching-based Online SFT for Text Spotting

- 结合SFT与GRPO,用匹配机制在线优化坐标预测。
- 在艺术文字数据集ATS上,识别与定位均超越现有方法。
- 适合需要强识别与精准定位的复杂文本场景研究者。
文字定位需兼顾准确识别与精确空间定位。当前专用检测器在自然场景中能生成紧密边界框,但在复杂或艺术字体上表现不佳;而多模态大模型虽有强大识别能力,定位性能较弱。为增强检测器的通用识别能力并最大化定位精度,我们探索了基于多模态大模型的监督微调(SFT)与基于组相对策略优化(GRPO)的强化学习微调。发现SFT在提升识别方面效果不如GRPO,而GRPO在检测方面又弱于SFT。为此提出联合训练策略SupGRPO,同时使用SFT与GRPO进行优化。SupGRPO设计特定奖励函数,并引入基于匹配的在线SFT,仅对坐标标记进行微调,缓解了GRPO的奖励稀疏问题,也避免了SFT的实例顺序依赖问题。为评估挑战性场景,我们构建了艺术文字定位数据集ATS。实验表明,SupGRPO在识别与定位上均有提升,达到更优性能。代码与数据集将开源于https://github.com/Psycho-9/SupGRPO。
原文摘要 · Abstract (English)
Text spotting requires both accurate text recognition and precise spatial localization. Current specialised spotters excel at predicting tight bounding boxes in natural scenes, but falter on complex or artistic text, whereas multimodal large language models (MLLMs) possess strong recognition capabilities yet remain weak at localisation. To equip the text spotter with general and powerful recognition capabilities and to maximize its localization ability, we explore two MLLM-based fine-tuning methods: Supervised Fine-Tuning (SFT) and reinforcement learning fine-tuning based on Group Relative Policy Optimisation (GRPO). An interesting finding is that SFT is less effective than GRPO at enhancing recognition, while GRPO is less effective than SFT at enhancing detection. To compensate for each other's shortcomings, we introduce a joint training strategy, SupGRPO, which simultaneously optimizes the model using both SFT and GRPO. SupGRPO employs the specially designed reward functions and develops a matching-based online SFT applied solely to coordinate tokens. It both mitigates the reward sparsity problem of GRPO and avoids the instance order dependency problem of SFT. To evaluate particularly challenging cases, we curate ATS, a dataset for artistic text spotting. Experiments demonstrate that SupGRPO improves both text recognition and detection, and attains superior performance. Our code and dataset will be released at https://github.com/Psycho-9/SupGRPO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。