用渐进强化奖励提升文档解析的精准度，解决高精度下学习信号弱的问题。
DocPO: Advancing Document Policy Optimization via Tailored Step-Aware Rewards
- 设计步进感知退火机制，逐步增强奖励曲线陡度
- 在多个文档元素上实现比传统方法更高的准确率
- 无需人工标注即可构建针对性奖励，适合文档处理研究者
文档解析的强化学习常依赖基于编辑距离（如树编辑距离）的参考奖励，但在高精度阶段因奖励区分度弱而难以优化：接近正确的输出得分相近，导致学习信号不足。本文提出步进感知退火（SAA），一种即插即用的奖励锐化机制，在训练过程中逐步增加奖励曲率，放大高质量样本间的细微差异，同时保持早期学习的稳定性。基于SAA，我们构建了DocPO框架，采用元素特定的、以编辑距离为锚点的参考奖励：文本使用归一化字符串编辑距离（NED），表格使用树编辑距离相似性（TEDS），公式采用混合评分+编辑距离奖励。在OmniDocBench和DocElemHard上的实验表明，SAA在不需额外人工监督的前提下，持续提升了各类文档元素上基于GRPO的强化学习性能。
原文摘要 · Abstract (English)
Reinforcement learning (RL) for document parsing often relies on reference-based rewards rooted in edit distance (e.g., tree edit distance), yet it remains hard to optimize in the high-accuracy regime because such rewards become weakly discriminative: near-correct outputs receive very similar scores, providing limited learning signal for hard cases. We propose Step-Aware Annealing (SAA), a plug-and-play reward sharpening mechanism that progressively increases reward curvature during training, amplifying subtle quality differences among high-scoring samples while preserving stability in early learning. Built on SAA, we introduce DocPO, a document policy optimization framework with element-specific, reference-based rewards anchored by edit-distance signals: normalized string edit distance (NED) for text, tree edit distance similarity (TEDS) for tables, and a hybrid Rubric+edit reward for formulas. Experiments on OmniDocBench and DocElemHard show that SAA consistently improves GRPO-style RL across document elements over non-annealed rewards, without requiring additional human supervision for reward construction.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。