DPO通过全局神经元激活调节降低毒性，而非仅抑制少数毒害神经元。
How Does DPO Reduce Toxicity? A Mechanistic Neuron-Level Analysis
- DPO通过分布式的多层神经元激活调整实现毒性抑制。
- 毒害神经元仅贡献2.5%至24%的去毒效果，其余来自整体激活平衡。
- 提出无需权重更新的激活编辑法，去毒更高效且保持模型困惑度。
安全微调算法能减少语言模型的有害输出，但其作用机制仍不清晰。直接偏好优化（DPO）是常用方法，但以往解释仅归因于MLP层中毒害神经元的抑制，不够完整。本研究分析了四个模型（Llama-3.1-8B、Gemma-2-2B、Mistral-7B、GPT-2-Medium），发现毒害神经元仅贡献2.5%至24%的去毒效果。实际上，DPO通过在所有MLP神经元间实现分布式激活偏移，产生净毒性下降。我们识别出四类神经元：两类促进降毒，两类增强抗毒能力，其协同效应可复现DPO效果。为进一步验证，我们开发一种激活编辑方法，沿毒性表示方向进行分布式调整，该方法在去毒上优于DPO，同时保持困惑度不变，且无需权重更新。本研究揭示了DPO的机制，并提出一种高效无调参的安全微调替代方案。
原文摘要 · Abstract (English)
Safety fine-tuning algorithms reduce harmful outputs in language models, yet their mechanisms remain under-explored. Direct Preference Optimization (DPO) is a popular choice of algorithm, but prior explanations, attributing its effects solely to dampened toxic neurons in the MLP layers, are incomplete. In this study, we analyse four language models (Llama-3.1-8B, Gemma-2-2B, Mistral-7B, GPT-2-Medium) and show that toxic neurons only account for 2.5% to 24% of DPO's effects across models. Instead, DPO balances distributed activation shifts across all MLP neurons to create a net toxicity reduction. We attribute this reduction to four neuron groups, two aligned with reducing toxicity and two promoting anti-toxicity, whose combined effects replicate DPO across models. To further validate this understanding, we develop an activation editing method mimicking DPO through distributed shifts along a toxicity representation. This method outperforms DPO in reducing toxicity while preserving perplexity, without requiring any weight updates. Our work provides a mechanistic understanding of DPO and introduces an efficient, tuning-free alternative for safety fine-tuning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。