对比强化学习与蒸馏，揭示大模型推理能力提升的关键机制
Reinforcement Learning vs. Distillation: Understanding Accuracy and Capability in LLM Reasoning
- RLVR聚焦简单题准确率，牺牲难题表现
- 蒸馏能同时提升准确率与能力，但非必然
- 响应质量提升不靠长度或关键词，需新指标
近期研究表明，基于可验证奖励的强化学习（RLVR）虽能提升大模型整体准确率（pass@1），却常无法改善推理能力（pass@k）。本文探究其机制：首先，RLVR因过度优化简单问题而损害对难题的准确率；其次，在小模型设置下，RLVR生成了原输出分布中不存在的高质量响应，且这些响应既不更长也无更多反思关键词，表明现有指标不可靠；最后，通过蒸馏教师模型回答发现，仅引入新知识时能力才提升，单纯蒸馏推理模式仅提高准确率但降低难题表现，类似RLVR。研究为理解两类方法如何塑造模型推理行为提供清晰洞见。
原文摘要 · Abstract (English)
Recent studies have shown that reinforcement learning with verifiable rewards (RLVR) enhances overall accuracy (pass@1) but often fails to improve capability (pass@k) of LLMs in reasoning tasks, while distillation can improve both. In this paper, we investigate the mechanisms behind these phenomena. First, we demonstrate that RLVR struggles to improve capability as it focuses on improving the accuracy of the easier questions to the detriment of the accuracy of the most difficult questions. Second, we show that RLVR does not merely increase the success probability for the easier questions, but in our small model settings, produces quality responses that were absent in its original output distribution. In addition, we show these responses are neither noticeably longer nor feature more reflection-related keywords, underscoring the need for more reliable indicators of response quality. Third, from the experiment distilling teacher responses to in-distribution problems, we find that capability does not always improve with distillation. We conjecture that capability improves only when new knowledge is introduced, whereas distilling reasoning patterns only improves accuracy but not capability, sacrificing performance on the most difficult questions, similar to RLVR. Together, these findings offer a clearer understanding of how RLVR and distillation shape reasoning behavior in LLMs
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。