用难例强化大模型安全，纠正有害推理链。
UnsafeChain: Enhancing Reasoning Model Safety via Hard Cases
- 从多样来源收集难例提示，强制纠正有害输出
- 1000条数据集即超越基线，显著提升安全表现
- 适合研究模型安全对齐与对抗性测试的团队
随着大模型推理能力增强，思维链（CoT）推理带来新的安全挑战。现有基于监督微调（SFT）的安全对齐研究主要聚焦于过滤能生成安全高质量回答的提示，忽略了那些始终引发有害输出的难例提示。为填补这一空白，我们提出UnsafeChain，一个由多种来源构成的难例安全对齐数据集，其中有害生成结果被识别并显式修正为安全响应。通过让模型暴露于有害行为并学习纠正，UnsafeChain在保持通用推理能力的同时增强安全性。我们在三款大推理模型上使用UnsafeChain进行微调，并在六个分布外和五个分布内基准上与近期SafeChain和STAR-1对比。结果表明，UnsafeChain持续优于先前数据集，甚至1000条子集的表现也达到或超过基线，证明了基于纠正的监督具有高效性与泛化性。数据集与代码已开源：https://github.com/mbzuai-nlp/UnsafeChain
原文摘要 · Abstract (English)
As large reasoning models (LRMs) grow more capable, chain-of-thought (CoT) reasoning introduces new safety challenges. Existing SFT-based safety alignment studies dominantly focused on filtering prompts with safe, high-quality responses, while overlooking hard prompts that always elicit harmful outputs. To fill this gap, we introduce UnsafeChain, a safety alignment dataset constructed from hard prompts with diverse sources, where unsafe completions are identified and explicitly corrected into safe responses. By exposing models to unsafe behaviors and guiding their correction, UnsafeChain enhances safety while preserving general reasoning ability. We fine-tune three LRMs on UnsafeChain and compare them against recent SafeChain and STAR-1 across six out-of-distribution and five in-distribution benchmarks. UnsafeChain consistently outperforms prior datasets, with even a 1K subset matching or surpassing baseline performance, demonstrating the effectiveness and generalizability of correction-based supervision. We release our dataset and code at https://github.com/mbzuai-nlp/UnsafeChain
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。