通过自动合成可验证数据，让大模型在知识密集领域推理更准且可追溯。
Knowledge-to-Verification: Exploring RLVR for LLMs in Knowledge-Intensive Domains
- 用自动化方法生成可验证的训练数据，扩展强化学习到知识密集领域。
- 模型在知识类任务中推理准确率提升，且通用能力基本不受影响。
- 适合研究大模型推理增强与可信AI的学者和开发者。
基于可验证奖励的强化学习（RLVR）在数学与编程等领域的大型语言模型（LLM）推理能力提升方面展现出潜力。然而，由于高质量可验证数据稀缺，其在知识密集型领域应用尚未有效探索。此外，现有RLVR仅关注最终答案正确性，导致推理过程存在缺陷且奖励信号稀疏。本文提出Knowledge-to-Verification（K2V）框架，通过自动化可验证数据合成，将RLVR拓展至知识密集型领域，并支持对模型推理过程的验证。大量实验表明，K2V在不显著损害模型通用能力的前提下，显著提升了LLM在知识密集任务中的推理表现。该研究还表明，结合自动化数据合成与推理验证是增强模型在更广泛领域能力的可行方向。代码已开源：https://github.com/SeedScientist/K2V。
原文摘要 · Abstract (English)
Reinforcement learning with verifiable rewards (RLVR) has demonstrated promising potential to enhance the reasoning capabilities of large language models (LLMs) in domains such as mathematics and coding. However, its applications on knowledge-intensive domains have not been effectively explored due to the scarcity of high-quality verifiable data. Furthermore, current RLVR focuses solely on the correctness of final answers, leading to the limitations of flawed reasoning and sparse reward signals. In this work, we propose Knowledge-to-Verification (K2V), a framework that extends RLVR to knowledge-intensive domains through automated verifiable data synthesis, while enabling verification of the LLM's reasoning process. Extensive experiments demonstrate that K2V enhances the reasoning of LLM in knowledge-intensive domains without significantly compromising the model's general capabilities. This study also suggests that integrating automated data synthesis with reasoning verification is a promising direction to enhance model capabilities in these broader domains. Code is available at https://github.com/SeedScientist/K2V.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。