实证研究如何通过微调提升大模型推理安全，发现简单方法更有效。
How Should We Enhance the Safety of Large Reasoning Models: An Empirical Study
- 通过数据蒸馏显式修复五类风险模式提升安全性能。
- 短流程或模板化推理可达到与复杂推理相当的安全效果。
- 适合关注大模型安全落地的研究者和工程师参考。
大型推理模型（LRMs）在数学、编程等需要深度推理的任务上表现卓越，但其增强的推理能力并不必然带来安全性的提升，甚至可能恶化。本文针对如何提升LRMs安全性展开全面的实证研究，聚焦监督微调（SFT）。研究发现，直接从DeepSeek-R1蒸馏安全回复无法显著改善安全性，我们分析其原因并识别出五种关键风险模式。通过在数据蒸馏阶段主动解决这些问题，实现了显著的安全性提升。进一步实验表明，复杂的长推理过程并非必要，简短或模板化的推理流程即可达到相近的安全水平。最后，我们通过系统的消融实验揭示了不同训练配置的影响。本研究为提升大模型安全性提供了更全面的实践视角。代码与数据已开源：https://github.com/thu-coai/LRM-Safety-Study。
原文摘要 · Abstract (English)
Large Reasoning Models (LRMs) have achieved remarkable success on reasoning-intensive tasks such as mathematics and programming. However, their enhanced reasoning capabilities do not necessarily translate to improved safety performance-and in some cases, may even degrade it. This raises an important research question: how should we enhance the safety of LRMs? In this paper, we present a comprehensive empirical study on how to enhance the safety of LRMs through Supervised Fine-Tuning (SFT). Our investigation begins with an unexpected observation: directly distilling safe responses from DeepSeek-R1 fails to significantly enhance safety. We analyze this phenomenon and identify five key risky patterns that contribute to it. We then demonstrate that explicitly addressing these issues during the data distillation process can lead to substantial safety improvements. Next, we explore whether a long and complex reasoning process is necessary for achieving safety. Interestingly, we find that simply using short or template-based reasoning process can attain comparable safety performance. These findings prompt a deeper reflection on the role of reasoning in ensuring safety. Finally, we conduct a comprehensive ablation study to reveal the impact of different training configurations. Overall, we hope our empirical study could provide a more holistic picture on enhancing the safety of LRMs. The code and data used in our experiments are released in https://github.com/thu-coai/LRM-Safety-Study.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。