用推理电路活跃度选强化学习训练题，效果比随机选好
CircuitLens: Reasoning Circuits as Data Selection Signals for Reinforcement Learning with Verifiable Rewards
- 通过分析46个敏感注意力头，快速计算问题推理强度得分
- 低活跃度题反而提升训练效果，最高活跃度组无优势
- 适合数学推理类强化学习，尤其在中等难度任务上
基于可验证奖励的强化学习（RLVR）对训练数据敏感，但现有选择标准将问题价值视为与模型无关的固有属性。本文提出电路推理得分（CRS），通过对比消融识别出46个推理敏感注意力头，在冻结基础模型上单次前向传播即可计算，无需奖励标签或仿真。实验显示，在Qwen2.5-Math-7B上，低活跃度题的十等分组在三个中等难度基准上优于随机选择（GSM8K +2.0 pp，OlympiadBench +1.6 pp，Minerva +2.9 pp），而高活跃度组收益更小且与中等组无显著差异。该优势存在边界条件：在领域精调数据池中各方法无区别；1.5B规模下有效方向不同；最低奖励训练条件产生最强泛化能力。在测试范围内，RLVR的数据选择呈现模式依赖性，而非静态问题质量排序。
原文摘要 · Abstract (English)
Reinforcement learning with verifiable rewards (RLVR) is sensitive to which problems a model trains on, yet existing selection criteria--difficulty filtering, hand-curation, reward-trajectory scoring--assess data value as an intrinsic property of problems, independent of the model that will learn from them. We introduce Circuit Reasoning Score (CRS), a selection signal derived from 46 reasoning-sensitive attention heads identified via contrastive ablation, computed in a single forward pass on the frozen base model without reward labels or rollouts. CRS runs against the intuitive hypothesis that stronger reasoning-circuit engagement produces better training data: on Qwen2.5-Math-7B, the lowest-engagement decile improves over random selection on three medium-difficulty benchmarks (GSM8K +2.0 pp, OlympiadBench +1.6 pp, Minerva +2.9 pp), while the highest-engagement decile gains less and is indistinguishable from the middle decile. The advantage has boundary conditions: on a domain-curated pool no selection method separates from the others; at 1.5B scale the useful direction differs; and the lowest-reward training condition produces the strongest downstream generalization. Within the Qwen2.5-Math settings tested, RLVR data selection appears regime-dependent rather than reducible to a static ranking of problem quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。