微调导致大模型安全对齐下降，且主流奖励模型不可靠。
Picky LLMs and Unreliable RMs: An Empirical Study on Safety Alignment after Instruction Tuning
- 分析三类影响安全对齐的因素：回答结构、身份校准、角色扮演。
- 发现良性微调后模型更易生成不当回复，安全性能下降。
- 揭示当前奖励模型无法准确反映人类安全偏好，适用性存疑。
大型语言模型（LLMs）在应对各类通用问题和任务中表现强大。然而，在较小的领域特定数据集上对已对齐的LLM进行微调，即使数据本身无害，也可能意外削弱其安全对齐能力，使模型更容易生成不恰当回应。本研究系统分析了良性微调场景下安全对齐退化的成因。分析识别出三个关键影响因素：回答结构、身份校准与角色扮演。此外，我们评估了当前最先进的奖励模型（RMs）的可靠性，这些模型常用于引导对齐过程。结果表明，这些RMs频繁无法准确反映人类对安全性的偏好，暴露出其在实际应用中的局限性。本工作揭示了微调过程中维持安全对齐的复杂性，并为开发者在实用性与安全性之间取得平衡提供了指导。实验所用数据集与微调代码可在 https://github.com/GuanlinLee/llm_instruction_tuning 获取。
原文摘要 · Abstract (English)
Large language models (LLMs) have emerged as powerful tools for addressing a wide range of general inquiries and tasks. Despite this, fine-tuning aligned LLMs on smaller, domain-specific datasets, critical to adapting them to specialized tasks, can inadvertently degrade their safety alignment, even when the datasets are benign. This phenomenon makes models more susceptible to providing inappropriate responses. In this study, we systematically examine the factors contributing to safety alignment degradation in benign fine-tuning scenarios. Our analysis identifies three critical factors affecting aligned LLMs: answer structure, identity calibration, and role-play. Additionally, we evaluate the reliability of state-of-the-art reward models (RMs), which are often used to guide alignment processes. Our findings reveal that these RMs frequently fail to accurately reflect human preferences regarding safety, underscoring their limitations in practical applications. By uncovering these challenges, our work highlights the complexities of maintaining safety alignment during fine-tuning and offers guidance to help developers balance utility and safety in LLMs. Datasets and fine-tuning code used in our experiments can be found in https://github.com/GuanlinLee/llm_instruction_tuning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。