通过概念消融微调，无需改数据就能控制大模型的泛化方向。
Steering Out-of-Distribution Generalization with Concept Ablation Fine-Tuning
- 利用隐空间概念投影，在微调时消去不良概念
- 使模型对泛化问题的回答错误率降低10倍
- 适合需要精准控制模型行为的研究者
大语言模型微调可能导致意外的分布外泛化。现有方法通常依赖修改训练数据以明确期望泛化，但并不总可行。本文提出概念消融微调（CAFT），利用可解释性工具在不修改训练数据或引入目标分布数据的情况下，控制模型的泛化路径。通过在线性投影中消除与不良概念对应的隐空间方向，实现对泛化的引导。在三个微调任务中验证成功，包括一种现象：在窄任务上微调后，模型对通用问题给出严重偏离对齐的回答。在不改变训练数据的前提下，CAFT将错误回答减少10倍，同时保持原分布性能。该方法为无需修改数据即可引导模型泛化提供了新路径。
原文摘要 · Abstract (English)
Fine-tuning large language models (LLMs) can lead to unintended out-of-distribution generalization. Standard approaches to this problem rely on modifying training data, for example by adding data that better specify the intended generalization. However, this is not always practical. We introduce Concept Ablation Fine-Tuning (CAFT), a technique that leverages interpretability tools to control how LLMs generalize from fine-tuning, without needing to modify the training data or otherwise use data from the target distribution. Given a set of directions in an LLM's latent space corresponding to undesired concepts, CAFT works by ablating these concepts with linear projections during fine-tuning, steering the model away from unintended generalizations. We successfully apply CAFT to three fine-tuning tasks, including emergent misalignment, a phenomenon where LLMs fine-tuned on a narrow task generalize to give egregiously misaligned responses to general questions. Without any changes to the fine-tuning data, CAFT reduces misaligned responses by 10x without degrading performance on the training distribution. Overall, CAFT represents a novel approach for steering LLM generalization without modifying training data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。