arXiv:2605.28388cs.AI2026-05被引 1

揭示难易样本对大模型强化学习的影响机制,提出自适应优化策略。

Mechanistically Interpreting the Role of Sample Difficulty in RLVR for LLMs

论文配图:Mechanistically Interpreting the Role of Sample Difficulty in RLVR for LLMs
图 1 · 摘自论文原文
  • 按样本难度分析强化学习中的推理信号,发现中等难度效果最佳。
  • 难样本易导致重复答案或跳步计算,反而损害模型能力。
  • 基于特征动态调整策略,提升硬样本利用效率,适合模型调优者。

强化学习结合可验证奖励(RLVR)在数学与编程任务中显著提升大语言模型的推理能力,但其对样本难度的机制仍不清晰。本文通过分难度与单样本分析发现:简单与中等难度问题带来最强且最稳定的推理改进;过于困难的问题常提供弱学习信号,引发重复回答或跳过必要计算等退化行为,甚至削弱模型原有能力。进一步使用时间稀疏自编码器(T-SAE)分析模型内部特征动态,发现:简单问题强化直接作答与基础计算特征,抑制深思型推理;难题激活推理相关特征,仅在成功轨迹被采样时有效;中等难度问题则平衡增强计算与多步推理特征。基于此,提出面向难样本的自适应策略,结合反向推理重构与T-SAE引导的训练信号,以提升奖励密度与信用分配效率。结果表明,样本难度是决定RLVR优化动态与表征演化的关键因素。

原文摘要 · Abstract (English)

Reinforcement Learning with Verifiable Reward (RLVR) is empirically shown to notably enhance the reasoning performance of large language models (LLMs), particularly in mathematics and programming. However, the mechanistic role of Sample Difficulty in RLVR remains poorly understood. In this paper, we investigate RLVR through the lens of difficulty-wise and one-sample analysis. We find that sample difficulty has a non-monotonic effect on RLVR: easy and medium-difficulty problems yield the strongest and most stable reasoning improvements, whereas overly hard problems often provide weak learning signals, induce degenerate behaviors such as answer repetition or skipping necessary computation, and can ultimately degrade the model's pre-existing capabilities. Beyond the obverse of response, we further analyze the model's internal feature dynamics using Temporal Sparse Autoencoders (T-SAE). Easy problems mainly reinforce direct-answer and basic-computation features while suppressing deliberative-reasoning features; hard problems activate reasoning-related features but become useful only when successful trajectories are sampled; medium-difficulty problems provide a more balanced signal, strengthening both computation and multi-step reasoning features. Motivated by these findings, we propose difficulty-adaptive strategies for hard-sample utilization, using backward-reasoning reformulation and T-SAE-guided training signals to improve reward density and credit assignment during RLVR. Overall, our results identify sample difficulty as a key factor governing both the optimization dynamics and representation evolution of RLVR.

强化学习大模型推理样本难度特征分析

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。