用可解释的指标捕捉大模型微调时的突然行为失准
Decomposing Behavioral Phase Transitions in LLMs: Order Parameters for Emergent Misalignment
- 通过分布变化检测与自然语言阶参数，追踪微调过程中的行为突变
- 发现实际行为转变比梯度峰值出现晚，且输出冗余占比超60%
- 适用于评估知识、政治、伦理等敏感场景下的模型对齐风险
在狭义有害数据集上微调大语言模型，可能导致其行为广泛偏离人类价值观。为理解这种涌现性失准的发生时机与机制，我们提出一个综合框架，利用分布变化检测方法和以自然语言表述的阶参数，结合大模型评判器，识别并表征微调过程中的快速转变。通过客观统计差异度量，量化了微调过程中相位转变对模型输出多方面的影响。特别地，我们评估了不同维度（如对齐度、冗长性）所贡献的分布变化占比，实现了整体转变的分解。研究发现，实际行为转变发生时间晚于梯度范数峰值指示的时间。该框架可自动发现并量化基于语言的阶参数，在从知识问答到政治伦理等多样示例中得到验证。
原文摘要 · Abstract (English)
Fine-tuning LLMs on narrowly harmful datasets can lead to behavior that is broadly misaligned with respect to human values. To understand when and how this emergent misalignment occurs, we develop a comprehensive framework for detecting and characterizing rapid transitions during fine-tuning using both distributional change detection methods as well as order parameters that are formulated in plain English and evaluated by an LLM judge. Using an objective statistical dissimilarity measure, we quantify how the phase transition that occurs during fine-tuning affects multiple aspects of the model. In particular, we assess what percentage of the total distributional change in model outputs is captured by different aspects, such as alignment or verbosity, providing a decomposition of the overall transition. We also find that the actual behavioral transition occurs later in training than indicated by the peak in the gradient norm alone. Our framework enables the automated discovery and quantification of language-based order parameters, which we demonstrate on examples ranging from knowledge questions to politics and ethics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。