不同微调方法影响模型拒绝有害请求的内部机制。
Beyond Shallow Alignment: How Post-Training Methods Determine Refusal Circuits And Steering Robustness
- 比较了三种后训练方法对拒绝机制的影响
- 推理增强训练产生稳定可识别的拒绝模式
- 现有方法无法同时实现安全、能力与可控性
语言模型如何拒绝有害请求，取决于后训练方法而非仅数据。我们对比了三种方法——监督微调、推理增强微调（基于解释链训练）和偏好优化（ORPO），在三个不同架构的模型（Llama-3.1-8B、Gemma-2-9B、Qwen3-8B）上进行测试。结果表明，训练方法显著重塑了拒绝机制的内部计算方式：推理增强训练在所有模型中均产生一致的拒绝模式；模型架构则独立影响内部结构及拒绝行为的可调控性。最重要的是，当前任一方法均未能同时满足三个理想特性：拒绝不集中在少数脆弱模块、安全提升不损害通用能力、安全行为可通过小范围精准修改调整。研究警示，不应将现有后训练方法视为可靠的安全防护方案，尤其在高安全要求场景下。代码与模型已开源于 https://github.com/hoangcuongnguyen2001/Beyond-Shallow-Alignment。
原文摘要 · Abstract (English)
How do the methods used to train language models to refuse harmful requests shape how that refusal actually works inside the model? We compare three post-training methods - supervised fine-tuning, reasoning-augmented fine-tuning (training on reasoning chains that justify a safety decision), and preference optimization (ORPO) - across three architecturally distinct models (Llama-3.1-8B, Gemma-2-9B, Qwen3-8B). We find that training method, not just data, reshapes how refusal is computed internally: reasoning-augmented training consistently produces a distinct kind of refusal computation, visible across all three models, while architecture independently shapes internal structure and how reliably refusal can be steered. Most importantly, no method we study achieves all three properties we would want from safe alignment at once: refusal that isn't concentrated in a few fragile components, safety gains that don't cost general capability, and safety behavior correctable through small, targeted edits. We caution against treating current post-training methods as a solved, reliable defense, especially for security-critical use. Code and models are available in https://github.com/hoangcuongnguyen2001/Beyond-Shallow-Alignment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。