通过推理轨迹的结构收敛性，可有效识别强化学习训练数据。
Detecting RLVR Training Data via Structural Convergence of Reasoning
- 基于生成结果的相似性差异，设计无模型检测方法。
- 在多个模型上准确区分已训练与未训练样本，准确率超基线。
- 适合关注模型数据泄露与评测污染的研究者使用。
强化学习结合可验证奖励（RLVR）是训练现代推理模型的核心方法，但其未公开的训练数据引发基准污染担忧。与基于词元概率优化的预训练不同，RLVR通过自生成推理轨迹的奖励反馈进行微调，使得传统基于似然的检测方法效果不佳。我们发现，RLVR会诱发独特的行为特征：训练中出现过的提示会产生更僵化、更相似的生成结果，而未见过的提示则保持更高多样性。为此提出Min-$k$NN Distance，一种无需访问参考模型或词元概率的黑盒检测器，通过为给定提示采样多个补全，并计算前$k$个最近邻的编辑距离平均值来量化这种退化。在多个RLVR训练的推理模型上的实验表明，该方法能可靠区分已见与未见样本，优于现有的成员推断与强化学习污染检测基线。
原文摘要 · Abstract (English)
Reinforcement learning with verifiable rewards (RLVR) is central to training modern reasoning models, but the undisclosed training data raises concerns about benchmark contamination. Unlike pretraining methods, which optimize models using token-level probabilities, RLVR fine-tunes models based on reward feedback from self-generated reasoning trajectories, making conventional likelihood-based detection methods less effective. We show that RLVR induces a distinctive behavioral signature: prompts encountered during RLVR training result in more rigid and similar generations, while unseen prompts retain greater diversity. We introduce Min-$k$NN Distance, a simple black-box detector that quantifies this collapse by sampling multiple completions for a given prompt and computing the average of the $k$ smallest nearest-neighbor edit distances. Min-$k$NN Distance requires no access to the reference model or token probabilities. Experiments across multiple RLVR-trained reasoning models show that Min-$k$NN Distance reliably distinguishes RL-seen examples from unseen ones and outperforms existing membership inference and RL contamination detection baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。