用模板化微调和规则奖励提升科学信息抽取的推理能力
Beyond path selection: Better LLMs for Scientific Information Extraction with MimicSFT and Relevance and Rule-induced(R$^2$)GRPO
- 用结构化推理模板进行两阶段微调，无需高质量思维链数据
- 结合相关性与规则奖励的R²GRPO使关系抽取超越基线模型
- 适合需要强推理与记忆能力的科学文本分析任务
以往研究指出，基于可验证奖励的强化学习（RLVR）训练的强大大语言模型仅能优化推理路径，无法提升数学任务中的推理能力；而通过蒸馏的监督微调（SFT）则可实现。本文从科学信息抽取（SciIE）视角出发，发现当前大语言模型及推理型模型在该任务上表现不如小型BERT模型，因科学信息抽取既需推理又需记忆。我们提出两阶段训练方法：1. MimicSFT，使用结构化推理模板，无需高质量思维链数据；2. 基于相关性与规则诱导奖励的R²GRPO。在多个科学信息抽取基准测试中，两种方法均有效提升推理能力。结合MimicSFT的R²GRPO在关系抽取上超越基线大模型与专用监督模型。代码已公开于https://github.com/ranlislz/R2GRPO。
原文摘要 · Abstract (English)
Previous study suggest that powerful Large Language Models (LLMs) trained with Reinforcement Learning with Verifiable Rewards (RLVR) only refines reasoning path without improving the reasoning capacity in math tasks while supervised-finetuning(SFT) with distillation can. We study this from the view of Scientific information extraction (SciIE) where LLMs and reasoning LLMs underperforms small Bert-based models. SciIE require both the reasoning and memorization. We argue that both SFT and RLVR can refine the reasoning path and improve reasoning capacity in a simple way based on SciIE. We propose two-stage training with 1. MimicSFT, using structured reasoning templates without needing high-quality chain-of-thought data, 2. R$^2$GRPO with relevance and rule-induced rewards. Experiments on scientific IE benchmarks show that both methods can improve the reasoning capacity. R$^2$GRPO with mimicSFT surpasses baseline LLMs and specialized supervised models in relation extraction. Our code is available at https://github.com/ranlislz/R2GRPO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。