提出成对测试协议,评估大模型在批量服务下的安全拒答鲁棒性。
A Paired Testing Protocol for Batch-Conditioned Refusal Robustness in LLM Serving

- 设计成对测试法,对比单个与批量请求的安全行为差异。
- 发现批量处理导致0.16%真实安全行为翻转,输出不稳定是主要脆弱点。
- 适合模型安全评测者和部署方,关注批量环境下的风险漏检。
语言模型的安全评估通常将服务配置视为固定背景,但批量条件在相同提示可能单独评估、同步批量或置于连续批调度器中时成为未验证的变量。本文整合四项基于实证的研究,提出成对测试协议:研究A结合本地发现、评分修正判别和真实批处理确认;研究B测试跨模型泛化;研究C测试连续批组合效应;研究D执行批次无关核消融。本地测试显示安全标签变化率(0.51%)高于能力标签(0.14%),但经63个候选行判别后仅17个为真实行为翻转,校正后全集翻转率为0.16%。15模型扩展发现无普遍安全-能力偏移:翻转接近平衡(0.94倍),对齐类型无显著关联(p=0.942, η²=0.033),输出不稳定性是最强脆弱指标(r=0.909,置信区间[0.65, 0.97])。在靶向核消融中,标准vLLM复现55个候选中22个标签翻转,启用VLLM_BATCH_INVARIANT=1后降至0/55;组合测试亦未见聚合效应(灵敏度4.7个百分点)。建议采用精确堆栈验证:在实际批处理环境下评估拒答行为,配对安全提示与能力对照,并将低频方向性翻转与整体零效应分开报告。
原文摘要 · Abstract (English)
Safety evaluations of language models often treat serving configuration as fixed background infrastructure, but batch condition is an untested treatment variable whenever the same prompt may be evaluated alone, in a synchronized batch, or inside a continuous-batching scheduler. We synthesize four artifact-backed studies into a paired testing protocol: Study A combines local discovery, scorer-corrected adjudication, and true-batching confirmation; Study B tests cross-model generalization; Study C tests continuous-batch composition; and Study D runs a batch-invariant-kernel ablation. The local test finds safety-label changes more often than capability-label changes (0.51% vs. 0.14%), but adjudication of 63 candidate rows leaves only 17 genuine behavioral flips, implying a corrected full-set rate of 0.16%. The 15-model extension finds no detectable universal safety-over-capability skew: flips are near parity (0.94x), alignment type has no detectable association ($p=0.942$, $η^2=0.033$), and output instability is the strongest tested fragility screen ($r=0.909$, bootstrap 95% CI [0.65, 0.97]). In the targeted kernel ablation, standard vLLM reproduces 22/55 label flips on current score-flip candidates, while enabling VLLM_BATCH_INVARIANT=1 reduces the same test to 0/55 flips; the composition test separately finds no aggregate effect at 4.7pp sensitivity. The testing recommendation is exact-stack validation: evaluate refusal at the served batch setting, pair safety prompts with capability controls, and report low-rate directional flips separately from aggregate null effects.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。