用简单LLM基线就能高效发现模型版本间的行为差异。
Simple LLM Baselines are Competitive for Model Diffing
- 用改进的LLM生成自然语言描述，对比不同模型版本行为差异。
- 在抽象性上优于稀疏自编码器方法，且性能相当。
- 适合关注模型迭代中隐性行为变化的研究者或工程师。
标准LLM评估仅测试评测者预设的能力或倾向，遗漏了模型版本间的行为变化或新兴的对齐偏差。模型差分（model diffing）通过自动识别系统性行为差异来弥补这一缺陷。现有方法包括基于LLM的自然语言描述生成和基于稀疏自编码器（SAE）的可解释特征识别。然而，尚无系统性比较，也缺乏统一评估标准。本文提出针对泛化性、有趣性和抽象层级等关键目标的评估指标，并用于对比现有方法。结果表明，改进后的LLM基线表现与SAE方法相当，且通常能揭示更抽象的行为差异。
原文摘要 · Abstract (English)
Standard LLM evaluations only test capabilities or dispositions that evaluators designed them for, missing unexpected differences such as behavioral shifts between model revisions or emergent misaligned tendencies. Model diffing addresses this limitation by automatically surfacing systematic behavioral differences. Recent approaches include LLM-based methods that generate natural language descriptions and sparse autoencoder (SAE)-based methods that identify interpretable features. However, no systematic comparison of these approaches exists nor are there established evaluation criteria. We address this gap by proposing evaluation metrics for key desiderata (generalization, interestingness, and abstraction level) and use these to compare existing methods. Our results show that an improved LLM-based baseline performs comparably to the SAE-based method while typically surfacing more abstract behavioral differences.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。