为文字识别设计2D旋转位置编码,更好处理弯曲和扭曲文本。
2D Rotary Position Embedding for Scene Text Recognition with Transformers

- 根据文本长宽比调整行列维度,适配非正方形文本区域。
- 将旋转编码扩展至跨注意力机制,支持解码器定位编码器中的2D布局。
- 在弯曲、旋转、透视扭曲文本上提升明显,适合复杂场景文字识别。
场景文字识别因文本形态多样(如弯曲、旋转、透视畸变)仍具挑战性。现有基于Transformer的方法多采用一维位置编码,忽略文本图像的二维结构。虽有轴向二维旋转位置编码(RoPE)用于视觉Transformer,但其假设图像内容近似正方形且仅在编码器自注意力中应用。而场景文字图像具有显著的各向异性,且识别模型为编码器-解码器结构,解码器需通过交叉注意力关联编码器的2D布局。本文提出2D-RoPE-STR,通过(1)按文本长宽比分配行/列维度,实现各向异性适配;(2)将旋转耦合扩展至编码器-解码器交叉注意力,使自回归解码步骤能依据2D布局关注编码器令牌。两项改进均无需额外参数或架构修改。我们还引入诊断协议(控制消融对、图像级预测差异分析、编码器注意力可视化),揭示相对2D位置在曲线、旋转及透视畸变布局中起作用,尤其在阅读顺序偏离水平直线时。在六个标准基准(IIIT5K, SVT, ICDAR 2013, ICDAR 2015, CUTE80, SVTP)上,性能增益集中于这些不规则布局,消融实验验证了每项设计优于1D RoPE、2D正弦与可学习位置编码。
原文摘要 · Abstract (English)
Scene Text Recognition (STR) remains challenging due to the diversity of text appearances, including curvature, rotation, and perspective distortion. Recent Transformer-based approaches perform well but usually rely on one-dimensional positional encodings that ignore the 2D spatial structure of text images. Axial 2D extensions of Rotary Position Embedding (RoPE) exist for vision Transformers, but they assume roughly square, isotropic image content and apply the rotation only within encoder self-attention. Scene text violates both assumptions: crops are markedly anisotropic, and STR models are encoder-decoder, so the decoder must relate its queries to the encoder's 2D layout through cross-attention. We introduce 2D-RoPE-STR, which adapts axial 2D-RoPE to this setting through (1) an anisotropic row/column dimension allocation matched to the aspect ratio of text, and (2) an extension of the rotary coupling into encoder-decoder cross-attention, letting autoregressive decoding steps attend to encoder tokens by their 2D layout, a setting not addressed by prior encoder-only formulations. Both changes are essentially parameter-free and require no architectural redesign beyond the positional-encoding module. We further introduce a diagnostic protocol (a controlled ablation pair isolating only the positional encoding, an image-level net-win disagreement analysis, and encoder attention visualization) that identifies where and why relative 2D position helps: curved, rotated, and perspective-distorted layouts where reading order departs from a straight horizontal line. On six standard benchmarks (IIIT5K, SVT, ICDAR 2013, ICDAR 2015, CUTE80, SVTP), gains concentrate on exactly these irregular layouts, with ablations isolating each design choice against 1D RoPE and 2D sinusoidal and learnable alternatives.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。