用维基共现统计替代复杂验证器，实现高效精准的问答奖励机制
Verifiable Rewards Beyond Math and Code: Lightweight Corpus-Grounded Process Supervision for Factual Question Answering
- 基于维基百科共现统计生成轻量级句子级奖励信号
- 在30个实验中平均提升TriviaQA准确率4.1个百分点
- 比神经验证器快4.8至8.4倍，适合大规模强化学习训练
将强化学习应用于提升知识密集型问答的事实准确性时面临奖励设计难题。响应级奖励监督粗糙，无法区分推理链中正确与错误陈述；句级奖励虽更精细，但通常依赖NLI验证器、大模型判断或知识验证流水线，部署成本高且对稀有实体事实可靠性差，而这些恰恰需要精确奖励信号。本文提出CorVer（Corpus Verify），一种轻量级、可即插即用的过程奖励机制，用维基百科共现统计替代神经验证器，为每个句子生成基于语料的可信度信号。CorVer通过简单对齐将句级奖励映射到词级别优势，仅需0.5B提取器和每句一次语料库查找。在涵盖六种指令微调模型（3B至14B）和五个QA基准的30个（模型，基准）组合中，所有情况下均优于原始基线，平均在TriviaQA上提升4.1个百分点。在20个可配置测试中，有18个超越四种神经验证器基线，训练速度提升4.8至8.4倍。
原文摘要 · Abstract (English)
Applying reinforcement learning to improve factual accuracy in knowledge-intensive question answering faces a reward design dilemma. Response-level rewards provide only coarse supervision and cannot distinguish correct from incorrect statements within a reasoning trace. Sentence-level alternatives offer finer-grained feedback, but typically rely on NLI verifiers, LLM judges, or knowledge-verification pipelines that are expensive to deploy at RL scale and often unreliable for rare-entity facts, where accurate reward signals are especially important. We propose CorVer (Corpus Verify), a lightweight, plug-in-ready process reward that replaces neural verifiers with a corpus-grounded signal derived from Wikipedia co-occurrence statistics. CorVer assigns sentence-level credit and maps it to token-level advantages via a simple alignment, requiring only a 0.5B extractor and a single corpus lookup per sentence. Across 30 (model, benchmark) cells spanning six instruction-tuned models (3B to 14B) and five QA benchmarks, CorVer improves over the raw baseline for every cell, with an average TriviaQA gain of +4.1 pp. It also outperforms four neural-verifier baselines in 18 of 20 cells under their feasible configurations, while training 4.8 to 8.4x faster.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。