分析大模型生成回复时风险如何变化,发现多数回复降低危害,少数反而升级。
From Prompt Risk to Response Risk: Paired Analysis of Safety Behavior of Large Language Models
- 对比人类标注的提示与回复,按危害类型和严重程度分级评估风险变化
- 61%回复降低危害,3%恶化,主要因提示本就危险或回复更严重
- 揭示有用性与安全性权衡,适合关注模型安全评测的研究者
大语言模型的安全评估通常只报告二元结果(如攻击成功率、拒绝率),忽略了提示与回复之间风险的变化。本文对四个危害类别(性、自残、仇恨、暴力)及四个严重等级(安全、低、中、高)的人类标注提示与回复对进行成对分析。结果显示,61%的回复降低了危害,36%保持原有严重程度,3%导致恶化。恶化分为两类:良性提示引发非预期的有害细节,以及在任务上保持但严重程度更高的回答。类别分解显示,性内容危害持续性最高,主要源于相同严重度下的顺从而非输入漂移。联合相关性分析揭示了有用性与安全性之间的权衡:顺从性恶化的回复仍高度相关,而安全回复多为通用拒绝且相关性低。对600个提示和六种模型的公开评估复现了该框架的测量结果与两个方向信号,而少样本模型评分器存在提示/回复检测不对称性,数据校准无法消除。代码与数据已开源。
原文摘要 · Abstract (English)
Safety evaluations of large language models (LLMs) typically report binary outcomes, i.e. attack success rate (ASR), refusal rate, or harmful versus safe classification, which hide how risk changes between prompt and response. We present a paired analysis over human labeled prompt and response records across four harm categories (Sexual, Self harm, Hate and Violence) and ordinal severity levels (Safe, Low, Medium, High). 61% of responses reduce harm relative to the prompt, 36% preserve severity, and 3% escalate. The escalation splits into two mechanisms: benign prompts triggering unrequested harmful detail, and answers that stay on task at higher severity than the prompt. Category decomposition shows that Sexual content exhibits the highest harm persistence in this sample, driven by compliance at the same severity rather than drift from benign inputs. Joint relevance analysis exposes a helpfulness versus harmlessness tradeoff: compliance escalations remain highly relevant, whereas safe responses include generic refusals with low relevance. A public supporting evaluation over 600 prompts and six models reproduces the framework's measurements and two directional signals, while few-shot LLM graders exhibit a prompt/response detection asymmetry that data calibration does not close. Grader prompts, public-evaluation artifacts, and analysis code are shared at https://github.com/microsoft/PairedSafety.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。