用任务特定剪枝加速路径修补，显著降低计算成本。
APP: Accelerated Path Patching with Task-Specific Pruning
- 引入对比注意力头剪枝，优先保留任务相关头以缩小搜索空间
- 相比原始方法提速59.63%至93.27%，平均减少56%搜索空间
- 适合需要高效电路发现的小模型或资源受限场景
电路发现是机制可解释性流程中的关键步骤。现有方法如路径修补计算开销大，且对小模型的深入分析能力有限。本文提出加速路径修补（APP），结合新型对比注意力头剪枝方法，大幅缩减电路发现的搜索空间。对比-FLAP剪枝算法借鉴因果中介分析思想，为任务相关注意力头分配更高剪枝得分，生成的稀疏模型性能优于传统剪枝技术。尽管对比-FLAP能有效保留低稀疏率下被移除的任务相关头，但其单独发现的电路仍过大，不满足最小性要求。APP先使用对比-FLAP将搜索空间平均压缩56%，再在剩余注意力头上应用传统路径修补，实现59.63%–93.27%的速度提升。尽管计算效率显著提高，但APP所得电路与已有路径修补结果高度重叠且性能相当。
原文摘要 · Abstract (English)
Circuit discovery is a key step in many mechanistic interpretability pipelines. Current methods, such as Path Patching, are computationally expensive and have limited in-depth circuit analysis for smaller models. In this study, we propose Accelerated Path Patching (APP), a hybrid approach leveraging our novel contrastive attention head pruning method to drastically reduce the search space of circuit discovery methods. Our Contrastive-FLAP pruning algorithm uses techniques from causal mediation analysis to assign higher pruning scores to task-specific attention heads, leading to higher performing sparse models compared to traditional pruning techniques. Although Contrastive-FLAP is successful at preserving task-specific heads that existing pruning algorithms remove at low sparsity ratios, the circuits found by Contrastive-FLAP alone are too large to satisfy the minimality constraint required in circuit analysis. APP first applies Contrastive-FLAP to reduce the search space on required for circuit discovery algorithms by, on average, 56\%. Next, APP, applies traditional Path Patching on the remaining attention heads, leading to a speed up of 59.63\%-93.27\% compared to Path Patching applied to the dense model. Despite the substantial computational saving that APP provides, circuits obtained from APP exhibit substantial overlap and similar performance to previously established Path Patching circuits
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。