通过比较模型输出概率差异,找出微调后新增的行为特征。
Diff Mining: Logit Differences Reveal Finetuning Objectives

- 对比微调前后模型的输出概率,定位关键变化的词元。
- 在无关文本上仍能识别出微调引入的核心行为特征。
- 适合用于检测模型偏见或验证微调目标,无需模型内部结构。
微调已成为改进语言模型行为、引入新能力的标准方法,但其过程中具体产生了哪些新行为仍不清晰。随着模型能力增强,理解微调过程愈发重要,尤其因为可能产生意外行为。本文提出Diff Mining,一种仅需访问输出logits的简单高效框架,通过比较微调模型与基础模型的logits差异,识别出被显著放大的关键词元,这些词元可作为微调痕迹的指纹,甚至在非微调领域文本中也能被发现。该框架分为两个模块:(i) 在参考语料上提取每上下文的logit差异;(ii) 通过Top-K频率法或基于非负矩阵分解(NMF)的方法聚合信号,分离出多个微调目标对应的独立词元簇。实验表明,该方法在检测微调领域方面显著优于现有模型差分方法,在下游任务中也表现更优;对于注入偏见的模型,无需针对性探测即可识别超过三分之一的偏见。整体表明该框架在开发微调目标审计工具方面具有潜力。
原文摘要 · Abstract (English)
Finetuning has become the gold standard for refining existing behaviors and inducing new ones in language models, yet it often remains unclear exactly which behaviors emerge during this process. As models grow ever more capable, understanding finetuning better becomes increasingly important, particularly since unwanted behaviors may arise during finetuning. In this paper, we introduce Diff Mining, a simple yet effective framework for identifying what a finetuned model has learned by comparing its logits to those of its base model. Diff Mining effectively surfaces salient tokens that are amplified in the finetuned model, serving as a fingerprint of its training -- even on text unrelated to the finetuning domain. Unlike many existing model diffing methods which require model internals, Diff Mining only needs access to output logits and scales to large models. The framework consists of two modular stages: (i) extracting per-context logit differences between the finetuned and base models on a reference corpus, and (ii) aggregating the resulting signals to construct an interpretable token set representing the finetune. For aggregation, we explore both a simple Top-K frequency method and a Non-negative Matrix Factorization (NMF)-based approach for disentangling multiple finetuning objectives into distinct token clusters. Empirically, Diff Mining succeeds across diverse settings: on finetune domain detection, it significantly outperforms state-of-the-art model diffing methods both in identifying relevant tokens and in downstream performance when an interpretability agent is given access to the extracted token set; on models with injected biases, it identifies more than one third of the biases without targeted probing. Overall, our framework shows promise in developing auditing tools to detect finetuning objectives.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。