用规则生成的合成数据训练大模型，显著提升多跳推理能力。
Learning from Synthetic Data Improves Multi-hop Reasoning
- 用规则生成虚构知识的合成数据进行强化学习微调。
- 在真实问答基准上表现优于原始模型，尤其在难题上进步明显。
- 适合需要低成本提升推理能力的研究者和开发者。
强化学习（RL）已被证明能显著提升大语言模型（LLM）在数学、编程和多跳推理任务中的推理能力。然而，RL微调需要大量高质量可验证数据，通常来自人工标注、前沿LLM生成或基于LLM的验证器评分，三者均有明显局限：人工标注数据量小且成本高，LLM生成数据易幻觉且昂贵，基于LLM的验证器准确率低且速度慢。本文探索了一种更廉价的替代方案：使用规则生成的合成数据对多跳推理任务进行强化学习微调。发现即使合成数据仅包含虚构知识，经其微调的LLM在主流真实世界问答基准上表现显著更好。按问题难度分层分析显示，合成数据教会了模型知识组合能力——一种基础且通用的推理技能。本工作表明，规则生成的合成推理数据是一种免费且可扩展的资源，可用于提升LLM的推理能力。
原文摘要 · Abstract (English)
Reinforcement Learning (RL) has been shown to significantly boost reasoning capabilities of large language models (LLMs) in math, coding, and multi-hop reasoning tasks. However, RL fine-tuning requires abundant high-quality verifiable data, often sourced from human annotations, generated from frontier LLMs, or scored by LLM-based verifiers. All three have considerable limitations: human-annotated datasets are small and expensive to curate, LLM-generated data is hallucination-prone and costly, and LLM-based verifiers are inaccurate and slow. In this work, we investigate a cheaper alternative: RL fine-tuning on rule-generated synthetic data for multi-hop reasoning tasks. We discover that LLMs fine-tuned on synthetic data perform significantly better on popular real-world question-answering benchmarks, despite the synthetic data containing only fictional knowledge. On stratifying performance by question difficulty, we find that synthetic data teaches LLMs to compose knowledge -- a fundamental and generalizable reasoning skill. Our work highlights rule-generated synthetic reasoning data as a free and scalable resource to improve LLM reasoning capabilities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。