让大模型通过自我反思推理，更安全地拒绝有害请求。
STAIR: Improving Safety Alignment with Introspective Reasoning
- 用带安全意识的思维链分析风险，而非直接拒绝
- 测试时缩放可达到与Claude-3.5相当的抗攻击能力
- 适合关注模型安全性与可用性平衡的研究者
确保大语言模型（LLMs）的安全性与无害性与其性能同样重要。然而，现有对齐方法常面临安全与性能的权衡，且易受越狱攻击，主要因其依赖对恶意查询的直接拒绝。本文提出STAIR框架，将安全对齐与内省推理结合。通过自提升的、具备安全意识的分步思维链（CoT）推理，使模型能识别安全风险。STAIR首先赋予模型结构化推理能力，再利用新提出的安全感知蒙特卡洛树搜索（SI-MCTS）生成逐步骤推理数据，通过迭代偏好优化推进对齐。此外，在该数据上训练过程奖励模型，以指导测试时搜索生成更优响应。大量实验表明，相较于本能对齐策略，STAIR有效减少有害输出并更好保持有用性。在测试时缩放下，其安全性表现媲美Claude-3.5，抵御主流越狱攻击。相关资源见https://github.com/thu-ml/STAIR。
原文摘要 · Abstract (English)
Ensuring the safety and harmlessness of Large Language Models (LLMs) has become equally critical as their performance in applications. However, existing safety alignment methods typically suffer from safety-performance trade-offs and the susceptibility to jailbreak attacks, primarily due to their reliance on direct refusals for malicious queries. In this paper, we propose STAIR, a novel framework that integrates SafeTy Alignment with Itrospective Reasoning. We enable LLMs to identify safety risks through step-by-step analysis by self-improving chain-of-thought (CoT) reasoning with safety awareness. STAIR first equips the model with a structured reasoning capability and then advances safety alignment via iterative preference optimization on step-level reasoning data generated using our newly proposed Safety-Informed Monte Carlo Tree Search (SI-MCTS). We further train a process reward model on this data to guide test-time searches for improved responses. Extensive experiments show that STAIR effectively mitigates harmful outputs while better preserving helpfulness, compared to instinctive alignment strategies. With test-time scaling, STAIR achieves a safety performance comparable to Claude-3.5 against popular jailbreak attacks. Relevant resources in this work are available at https://github.com/thu-ml/STAIR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。