用自动测试发现强化学习奖励验证器的漏洞，避免模型学坏。
Before the Model Learns the Bug:Fuzzing RLVR Verifiers
- 设计轻量级模糊测试框架，生成对抗性输出测试验证器
- 发现验证器错误导致模型优化时误学缺陷
- 适合关注RLVR安全性和验证器可靠性的研究者
基于可验证奖励的强化学习（RLVR）用可执行的奖励函数（如数学答案检查器、JSON工具调用验证器、代码单元测试框架）替代人类偏好标签。这使得奖励部分成为软件产物：若验证器存在缺陷，优化过程可能习得该错误。本文提出一种轻量级验证器模糊测试框架，可生成对抗性补全结果，对比有缺陷与更严格参考验证器的判断，记录成对决策，并报告误报、漏报、不一致、可利用漏洞及不确定性等指标。
原文摘要 · Abstract (English)
Reinforcement learning with verifiable rewards (RLVR) replaces human preference labels with executable reward functions such as math answer checkers, JSON tool-call validators, and code unit-test harnesses. That makes the reward partly a software artifact: if the verifier is wrong, optimization can learn the bug. We study this failure mode with a lightweight verifier-fuzzing framework that generates adversarial completions, compares buggy and stricter reference verifiers, logs paired decisions, and reports false-positive, false-negative, disagreement, exploit, and uncertainty metrics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。