模型架构比规模更重要，分组查询注意力让电路更集中稳定。
Architecture, Not Scale: Circuit Localization in Large Language Models
- 用分组查询注意力替代标准多头注意力，提升电路可解释性。
- 在相同规模下，分组查询注意力的电路集中度更高，稳定性更强。
- 适合关注模型内部机制、想理解大模型可解释性的研究者。
机械可解释性认为模型规模越大，电路分析越难。我们挑战这一假设，发现注意力架构的影响远超参数量。通过研究Pythia和Qwen2.5中的三类电路，发现分组查询注意力在相似规模下产生的电路更集中、机制更稳定。该集中模式在间接宾语识别、归纳头和事实回忆任务中均成立。在单一架构族（Qwen2.5）中，事实回忆电路在超过临界规模后发生离散相变，收缩为单一瓶颈而非渐进退化。结果表明，某些架构选择可使大模型更易研究，且可解释性难度并非模型规模的必然后果。
原文摘要 · Abstract (English)
Mechanistic interpretability assumes that circuit analysis becomes harder as models scale. We challenge this assumption by showing that the attention architecture matters more than parameter count. Studying three circuit types across Pythia and Qwen2.5, we find that grouped query attention produces circuits that are far more concentrated and mechanistically stable than standard multi-head attention at comparable scales. The same concentration pattern holds across indirect object identification, induction heads, and factual recall. Within a single architecture family (Qwen2.5), factual recall circuits undergo a discrete phase transition above a critical scale, collapsing to a single bottleneck rather than degrading gradually. These findings suggest that some architectural choices make large models more tractable to study and that interpretability difficulty is not a fixed consequence of model size.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。