通过预览部分答案，让大模型微调更安全
LookAhead Tuning: Safer Language Models via Partial Answer Previews
- 用部分答案前缀修改训练数据，减少对初始词分布的扰动
- 微调后仍保持高安全性，下游任务性能不下降
- 适合需要安全性的模型定制场景，如医疗、金融
微调能使大语言模型适应特定领域，但常导致原有安全对齐能力下降。为缓解微调过程中的安全退化问题，我们提出轻量级且高效的数据驱动方法——前瞻微调（LookAhead Tuning）。该方法通过在训练数据中引入部分答案前缀，以最小化对模型初始词分布的扰动，从而维持其内置的安全机制。大量实验表明，前瞻微调能有效保持模型安全性，同时不牺牲下游任务的鲁棒性能。研究结果表明，该方法是大模型安全高效适配的可靠方案。
原文摘要 · Abstract (English)
Fine-tuning enables large language models (LLMs) to adapt to specific domains, but often compromises their previously established safety alignment. To mitigate the degradation of model safety during fine-tuning, we introduce LookAhead Tuning, a lightweight and effective data-driven approach that preserves safety during fine-tuning. The method introduces two simple strategies that modify training data by previewing partial answer prefixes, thereby minimizing perturbations to the model's initial token distributions and maintaining its built-in safety mechanisms. Comprehensive experiments demonstrate that LookAhead Tuning effectively maintains model safety without sacrificing robust performance on downstream tasks. Our findings position LookAhead Tuning as a reliable and efficient solution for the safe and effective adaptation of LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。