提出先剪枝后解释的框架，显著降低电路发现成本。
Prune, Interpret, Evaluate: A Cross-Layer Transcoder-Native Framework for Efficient Circuit Discovery via Feature Attribution
- 先剪枝再解释，用梯度加权写入贡献评估特征重要性。
- 在K=50时达成基线K=75的性能，节省33%解释成本。
- 适合追求高效模型解释与低资源电路发现的研究者。
现有特征解释流程通常对均匀采样单元或全特征集进行处理，导致大量无关单元产生高额开销。为此，我们提出首个面向跨层转换器（CLT）的端到端剪枝框架PIE，开创了先剪枝后解释的新范式。PIE整合剪枝、自动解释与解释评估，建立全面基准环境，系统衡量剪枝下的行为保真度与下游可解释性。在此框架中，我们适配强相关性基线并提出特征归因补丁（FAP），一种基于补丁的归因方法，通过聚合梯度加权写入贡献来评分CLT特征。此外，引入FAP-Synergy，一种系统性的协同感知重排序机制。我们在IOI和Doc-String数据集上使用KL散度衡量行为保留率，以FADE风格指标评估解释质量。在预算约束K ∈ {50, 100, 200, 400, 800}下，基准测试揭示不同运行区间：在宽松预算下基线表现稳健，而FAP-Synergy在严格预算下表现更优。关键发现：在Llama-3.2-1B与Gemma-2-2B的IOI任务中，FAP-Synergy在K=50时功能等价于基线在K=75的表现，实现‘有效预算’优势——解释成本每特征线性增长，使该管道获得25个‘免费’特征，以33%成本降低达到同等精度。
原文摘要 · Abstract (English)
Existing feature-interpretation pipelines typically operate on uniformly sampled units or exhaustive feature sets, incurring massive costs on units irrelevant to target behaviors. To address this, we introduce the first CLT-native end-to-end pruning framework, PIE, which pioneers the paradigm of pruning first and interpreting later. PIE connects Pruning, automatic Interpretation, and interpretation Evaluation, establishing a comprehensive benchmarking environment to systematically measure behavioral fidelity and downstream interpretability under pruning. Within this framework, we adapt strong relevance baselines and propose Feature Attribution Patching (FAP), a patch-grounded attribution method that scores CLT features by aggregating gradient-weighted write contributions. Furthermore, we introduce FAP-Synergy, a systematic synergy-aware reranking procedure. We evaluate pruning using KL-divergence behavior retention and assess interpretation quality with FADE-style metrics across IOI and Doc-String datasets. Across budget constraints of K in {50, 100, 200, 400, 800}, our rigorous benchmarking reveals distinct operational regimes: while base FAP and adapted baselines perform robustly at relaxed budgets, FAP-Synergy excels in highly constrained, strict-budget regimes. Crucially, we demonstrate a practical "Effective Budget" advantage: on the IOI task for both Llama-3.2-1B and Gemma-2-2B, FAP-Synergy at K=50 functionally matches the behavioral fidelity of baseline circuits at K=75. Because downstream evaluation costs scale linearly per feature, Synergy effectively grants the pipeline 25 "free" features, achieving K=75 fidelity while reducing interpretation costs by 33%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。