arXiv:2604.24302cs.CL2026-04被引 2

用可微对齐方法,把小模型的电路知识迁移到大模型中。

Differentiable Faithfulness Alignment for Cross-Model Circuit Transfer

论文配图:Differentiable Faithfulness Alignment for Cross-Model Circuit Transfer
图 1 · 摘自论文原文
  • 通过可微映射将小模型节点重要性投影到大模型
  • 在Llama-3 1B→3B上表现接近直接归因,零样本迁移有效
  • 适合研究模型间机制迁移,但大模型差距下效果下降

机制可解释性已能定位语言模型中特定行为的电路,但现有方法成本高、依赖具体模型且难以扩展至更大架构。本文提出可微忠实度对齐(DFA),通过学习一个可微映射,将小型源模型的电路信息迁移到大型目标模型。DFA将源模型节点重要性得分投影至目标模型,并以软忠实度目标训练该映射,避免在目标模型上进行完整的电路发现。我们在Llama-3和Qwen-2.5上评估了六项任务,涵盖事实检索、多选推理与算术。最强结果出现在Llama-3 1B→3B设置中,对齐后的电路常与直接节点归因相当,零样本迁移仍有效。当源-目标模型差距增大时恢复能力减弱,且在Qwen-2.5上显著降低,表明架构与缩放差异越大,迁移越困难。总体而言,DFA持续优于简单基线,在某些场景下恢复的目标模型电路忠实度甚至超过或媲美直接归因。结果表明,小模型可为大模型提供有效的机制先验,同时揭示了节点级跨模型电路对齐的潜力与局限。

原文摘要 · Abstract (English)

Mechanistic interpretability has made it possible to localize circuits underlying specific behaviors in language models, but existing methods are expensive, model-specific, and difficult to scale to larger architectures. We introduce \textbf{Differentiable Faithfulness Alignment (DFA)}, a framework that transfers circuit information from a smaller source model to a larger target model through a learned differentiable alignment. DFA projects source-model node importance scores into the target model and trains this mapping with a soft faithfulness objective, avoiding full circuit discovery on the target model. We evaluate DFA on Llama-3 and Qwen-2.5 across six tasks spanning factual retrieval, multiple-choice reasoning, and arithmetic. The strongest results occur on Llama-3 $1$B$\rightarrow3$B, where aligned circuits are often competitive with direct node attribution and zero-shot transfer remains effective. Recovery weakens for larger source--target gaps and is substantially lower on Qwen-2.5, suggesting that transfer becomes harder as architectural and scaling differences increase. Overall, DFA consistently outperforms simple baselines and, in some settings, recovers target-model circuits with faithfulness comparable to or stronger than direct attribution. These results suggest that smaller models can provide useful mechanistic priors for larger ones, while highlighting both the promise and the limits of node-level cross-model circuit alignment.\footnote{Code is available at https://github.com/jasonshaoshun/dfa-circuits.

可解释性模型迁移机制分析

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。