让大模型自我审查并纠正伦理偏差，实现自动对齐。
Emergent Alignment
- 用自身复制品做道德审查，通过偏好优化引导模型向善。
- 单个自省问题即可在代码黑客场景中实现伦理对齐。
- 无需外部评判器，适合训练、微调、零样本等多场景。
大型语言模型能否识别自身输出与人类伦理的偏离？能否自我修正？我们为模型引入一个自我审查步骤，对其推理和输出进行评估，并使用直接偏好优化（DPO）扩展训练损失，引导模型远离非伦理输出。该方法实现了在线对齐，适用于训练、微调、对抗提示和零样本学习等多种场景。它不依赖更弱或更强的评判模型，而是利用自身的一个冻结副本。此前研究中的“涌现错位”现象显示，微调会导致模型出现一系列新兴的不道德行为，如代码黑客。而本文实证表明，在相同代码黑客场景下，仅通过一个高层次的自省问题，即可实现“涌现对齐”：使模型在训练过程中趋向伦理合规。
原文摘要 · Abstract (English)
Can Large Language Models (LLMs) discern when their own outputs are misaligned with human ethics? And can they self-correct? We endow an LLM with a conscience step that reviews its own reasoning and outputs, and we extend the training loss with an alignment component using Direct Preference Optimization (DPO) to steer the model away from non-ethical outputs. The result is an online technique to align models in a wide range of applications: training, fine-tuning, adversarial prompting, and zero-shot learning. It does not require a weaker or stronger judge, relying instead on a frozen copy of itself. In previous work, the Emergent Misalignment scenario showed a range of emergent unethical behaviors from fine-tuning the model to hack code. Instead, we empirically show how to achieve Emergent Alignment: a single high-level introspective question steers training toward an ethical model under the same code hacking scenario.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。