为强化学习训练的专用大模型提供实时安全验证，确保每轮部署都符合误差预算。
Conformal Selective Acting: Anytime-Valid Risk Control for RLVR-Trained LLMs
- 设计了一种逐轮验证机制，基于可验证奖励微调的模型实时生成安全证书。
- 在480个基准、160个对抗性分布偏移和超1万轮在线更新中均保持有效验证。
- 适合需实时合规的金融、医疗等强监管场景，不依赖前沿API接口。
在受监管机构中部署一个基于本地数据通过可验证奖励强化学习（RLVR）微调的局部专用大模型，每部署一轮都有固定的误差预算α。操作方需要在每轮即时获得安全认证：不能跨部署聚合数据，也不能等待长期平均结果。现有封装方法无法满足自适应在线更新流的要求：离线置信区间方法要求数据交换性；在线方法仅约束长期平均风险；非交换性扩展仅近似有效；最接近的即时方法A-RCPS控制的是边际风险而非选择性风险。本文提出置信选择行动（CSA），采用测试统计量、有效性保证与部署规则框架，填补了这一空白：对每个阈值维护维尔型e过程，在博纳费罗尼网格上评估，基于RLVR滤网。在可预测更新和等距校准单调风险条件下，证明了：(i) 任意时间路径上的选择性风险边界为 $R_T^{ ext{act}}\≤α+O(N_T^{-1/2})$，(ii) 认证速率最优，匹配 $Θ(\barη^{-2}\log(1/δ))$，(iii) 横向释放率差距与时间无关。在八类专业基准（480条流）、十六组对抗性分布偏移（160条流）及五组真实专家迭代RLVR实验（在线LoRA，四基模型，三架构族，共10,300轮）中，十种对比方法里唯独CSA满足路径有效性且不拒绝任何一轮部署。本文未提出新模型、训练算法或策略类别；CSA是部署端补充，与模型无关，适用于无法使用前沿API的操作方。
原文摘要 · Abstract (English)
A local specialist LLM, fine-tuned with reinforcement learning from verifiable rewards (RLVR) on operator-local data, is installed in a regulated organization with per-deployment error budget $α$. The operator needs a safety certificate for this deployment's stream at every round: no pooling across deployments, no waiting for a long-run average. Existing wrappers cannot deliver this on adaptive, online-updated streams: offline conformal-risk methods require exchangeability; online-conformal methods bound only long-run averages; non-exchangeable extensions are marginally valid; and the closest anytime wrapper, A-RCPS, controls marginal rather than selective risk. Using a (test statistic, validity guarantee, deployment rule) framework, we identify one empty cell forced by deployment requirements: e-process per threshold, selective risk, anytime-pathwise validity, max-certified-threshold rule. Conformal Selective Acting (CSA) fills it as a per-round wrapper maintaining a Ville-type e-process per threshold on a Bonferroni grid, evaluated against the RLVR filtration. Under predictable updates and isotonic-calibrated monotone risk we prove (i) an anytime-pathwise selective-risk bound $R_T^{\mathrm{act}}\leα+O(N_T^{-1/2})$, (ii) rate-optimal certification matching $Θ(\barη^{-2}\log(1/δ))$, and (iii) a horizon-independent release-rate gap. Across eight specialist benchmarks ($480$ streams), sixteen adversarial distribution-shift cells ($160$ streams), and five live Expert-Iteration RLVR cells with online LoRA over four base models in three architecture families ($10{,}300$ rounds), CSA is the only method among ten compared that satisfies pathwise validity and non-refusing deployment on every cell. We do not propose a new LLM, training algorithm, or policy class; CSA is the deployment-side complement, orthogonal to the model, for operators who cannot use a frontier API.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。