发现大模型推理训练中存在无法学习的难题，即使有正确答案也难纠正。
The Unlearnability Phenomenon in RLVR for Language Models
- 通过梯度分析发现，部分难题样本因表示缺陷而难以学习。
- 即使有正确推理路径，仍约30%的硬样本无法优化提升。
- 适合研究强化学习推理机制与模型瓶颈的学者参考。
基于可验证奖励的强化学习（RLVR）在提升大语言模型推理能力方面已证明有效，但其学习动态尚不明确。本文揭示了一种反直觉现象：在初始表现不佳的硬样本中，即使存在正确推理轨迹，仍有相当比例（约30%）始终无法被模型学会。为理解该现象，我们首先证明现有优化与采样方法无法解决此类不可学性。通过跨样本梯度分析，发现这些不可学习样本具有根本性的表示缺陷，表现为与其他样本梯度相似度低且推理模式无法泛化。进一步表明，在强化学习中，数据增强也无法改善梯度相似性。本研究首次系统刻画了RLVR训练中的不可学习数据，并揭示了当前强化学习方法在推理任务上的根本局限性。代码与数据见：https://github.com/yulinchen99/unlearnability-rlvr。
原文摘要 · Abstract (English)
Reinforcement Learning with Verifiable Reward (RLVR) has proven effective in improving Large Language Model's (LLM) reasoning ability. However, the learning dynamics of RLVR remain underexplored. In this paper, we reveal a counterintuitive phenomenon: among hard examples that the model initially struggles with, a substantial subset remains unlearnable even when correct rollouts are present. To understand the phenomenon, we first demonstrate that existing optimization and sampling techniques fail to resolve unlearnability. With cross-example gradient analysis, we show that unlearnable examples have fundamental representation issue, characterized by low gradient similarity with the rest of the examples and ungeneralizable reasoning patterns. We further show that representation flaws are difficult to mitigate in RL, as data augmentation does not improve gradient similarity. Our study provides the first systematic characterization of unlearnable data in RLVR training and reveals fundamental limitations in current RL approaches for reasoning tasks. Code and data are available at \url{https://github.com/yulinchen99/unlearnability-rlvr}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。