arXiv:2510.23264cs.LGcs.AI2025-10被引 5

通过混合精度优化,让电路发现提速80%且省30%内存。

PAHQ: Accelerating Automated Circuit Discovery through Mixed-Precision Inference Optimization

  • 对每个注意力头独立做精度优化,只在关键部分保持高精度。
  • 相比原方法,运行时间减少80%,内存降低30%且分析更准确。
  • 无需训练,可无缝接入现有电路发现工具,适合研究解释性的人。

电路发现旨在从预训练语言模型中识别稀疏且任务相关的子网络,是机制可解释性的核心。自动化电路发现(ACDC)虽重要,但因计算效率低和内存需求过高,难以应用于大模型。现有加速方法多依赖线性近似,严重牺牲分析准确性。本文提出的每注意力头量化(PAHQ)方法,通过激活插补与混合精度量化(MPQ)间的本质关联——插补实质为定向消融实验——仅对被研究组件保持高精度,其余部分可安全降精度。PAHQ加速的ACDC相较未加速版本,运行时间最多减少80%,内存消耗最多降低30%,同时保证分析忠实度。该方法无需训练,只需修改注意力计算机制,即可与现有基于边缘的电路发现技术集成,为机制可解释性提供高效新路径。代码已开源:https://github.com/626619403/PAHQ。

原文摘要 · Abstract (English)

Circuit discovery, which involves identifying sparse and task-relevant subnetworks in pre-trained language models, is a cornerstone of mechanistic interpretability. Automated Circuit Discovery (ACDC) has emerged as a pivotal methodology in circuit discovery, but its application to large language models is severely limited by computational inefficiency and prohibitively high memory requirements. Although several accelerated approaches have been proposed, they primarily rely on linear approximations to ACDC, which significantly compromises analytical faithfulness. Our proposed method for accelerating automated circuit discovery, Per Attention Head Quantization (PAHQ), takes a fundamentally different approach by optimizing the efficiency of each individual patching operation. PAHQ leverages a fundamental alignment between activation patching and mixed-precision quantization (MPQ): interpretability analysis through patching essentially performs targeted ablation studies. Therefore, we can maintain high precision exclusively for investigated components while safely reducing precision elsewhere in the network. PAHQ-accelerated ACDC reduces runtime by up to 80\% and memory consumption by up to 30\% compared to unaccelerated ACDC while maintaining faithfulness. Importantly, our method readily integrates with existing edge-based circuit discovery techniques by modifying the attention computation mechanism. This training-free approach provides a practical and novel pathway for accelerating mechanistic interpretability methods. Our code is available at https://github.com/626619403/PAHQ.

电路发现可解释性混合精度大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。