不训练、无标签,用隐藏状态变化选最优推理数据
Single-Rollout Hidden-State Dynamics for Training-Free RLVR Data Selection

- 仅靠一次推理的隐藏状态变化量,衡量样本价值
- 在极低预算下提升数学与医疗问答准确率
- 适合无标签、无法提前训练的领域数据筛选
强化学习中可验证奖励(RLVR)能在极少训练样本下带来显著推理提升,但其对数据敏感,数据选择成为关键瓶颈。现有方法多依赖训练时信号或大规模候选池的可验证奖励,成本高且难用于专业领域。本文提出SHIFT,一种无需训练、无需标签的一次性数据选择器。对每个候选样本,运行一次确定性推理流程,计算起始到结束的隐藏状态变化量(RIRS)作为样本效用的轻量代理。基于RIRS增强特征空间,采用质量加权的最远点优先聚类(CoreSet)实现覆盖,生成紧凑子集,可扩展至大规模未标注池。在数学推理与医疗问答基准上,超低预算下性能持续优于无训练的多样性及难度/不确定性基线,提升域内准确率并增强对更难测试场景的迁移能力。消融实验表明,RIRS覆盖与质量加权贡献互补;分析显示RIRS非由输入输出长度等简单统计解释。代码已开源。
原文摘要 · Abstract (English)
Reinforcement learning with verifiable rewards (RLVR) can yield large reasoning gains from very few training instances, yet its strong sensitivity to which instances are used makes data selection a central bottleneck. Most existing selection pipelines rely on training-time optimization signals and/or require access to verifiable rewards or ground-truth answers over large candidate pools, which is costly and often infeasible in specialized domains. We study RLVR data selection in a setting where selection must be performed before any RL training and without labels or reward evaluation on the full pool. We propose SHIFT, a one-shot, training-free selector based solely on inference-time hidden-state dynamics. For each candidate instance, SHIFT runs a single deterministic reasoning rollout and computes a reasoning-induced representation shift (RIRS) as the start-to-end hidden-state delta. SHIFT uses the RIRS magnitude as a lightweight proxy for instance utility and enforces coverage via a quality-weighted farthest-first CoreSet procedure in an RIRS-augmented feature space, producing compact subsets that scale to large unlabeled pools. Across mathematical reasoning and medical QA benchmarks under ultra-low budgets, SHIFT consistently outperforms training-free diversity and difficulty/uncertainty baselines, improving both in-domain accuracy and transfer to harder evaluation settings. Ablations show that RIRS-based coverage and quality-weighting contribute complementary gains, and analyses indicate that RIRS is not explained by simple input/output length statistics. Code is available at github.com/JianghaoWu/SHIFT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。