用可解释的稀疏自编码器选数据，提升大模型推理训练效率
IRDS: Interpretable RLVR Data Selection via Verifier-Coupled Sparse Autoencoder Coverage
- 基于稀疏自编码器聚类选择训练数据，可识别问题模式
- 在模型既失败又可学的数据上优化，准确率最高提升4.0个百分点
- 兼顾可解释性与低成本，适合需要透明训练的场景
基于可验证奖励的强化学习（RLVR）已成为提升大语言模型推理能力的关键技术，但其数据效率低仍是主要瓶颈。现有方法仅部分解决子集覆盖、验证器信号利用或可解释性中的问题。为此，我们提出IRDS（可解释的RLVR数据选择），基于稀疏自编码器（SAE）聚类选择训练实例，使选择过程可审计且对应可识别的问题模式。为选出模型失败但仍可学习的实例，我们在SAE基础上引入耦合验证器的覆盖率目标，并通过贪心对数行列式最大化求解。在三个指令微调模型和六个数学推理基准上的实验表明，IRDS实现最高整体准确率，在两个Qwen模型上分别超越最强基线+3.9/+4.0个百分点，在Llama-3.1-8B上提升+0.5个百分点，同时运行成本比基于轨迹的基线低一个数量级。
原文摘要 · Abstract (English)
Reinforcement learning with verifiable rewards (RLVR) has become a key technique for en- hancing LLM reasoning, yet its data ineffi- ciency remains a major bottleneck. Existing methods address this problem only partially, each missing at least one of subset-level cov- erage, verifier signal use, or interpretability. To address this gap, we present IRDS (Inter- pretable RLVR Data Selection), which selects RLVR training instances on a sparse autoen- coder (SAE) cluster basis so the selection itself is auditable on recognizable problem motifs. To select instances the model both fails on and can still learn from, we introduce a verifier- coupled coverage objective on the SAE basis and solve it by greedy log-determinant max- imization. Experiments on three instruction- tuned models and six math reasoning bench- marks show that IRDS achieves the highest overall accuracy, exceeding the strongest base- line by +3.9/+4.0 pp on the two Qwen models and by +0.5 pp on Llama-3.1-8B, while run- ning an order of magnitude cheaper than the trajectory-based baseline.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。