用激活统计快速定位Transformer中的推理电路，提速超千倍。
CircuitProbe: Predicting Reasoning Circuits in Transformers via Stability Zone Detection
- 通过梯度与异常评分检测早期/晚期推理电路位置
- 10个校准样本即可预测，9个模型验证准确率100%
- 适合小模型（<3B）的推理优化，跨语言稳定
Transformer语言模型中存在局部推理电路，即在推理时复制连续层块可提升推理能力。当前寻找这些电路需暴力扫描，每模型耗时25 GPU小时。本文提出CircuitProbe，仅需不到5分钟CPU时间，基于激活统计预测电路位置，速度提升三到四个数量级。我们发现推理电路分两类：早期层的稳定性电路通过表示变化导数检测，晚期层的幅度电路通过异常评分检测。在9个涵盖6种架构的模型（包括2025年模型）上验证，CircuitProbe最优预测结果与真实最佳电路位置匹配或仅差2层。对Qwen 2.5系列的缩放实验表明，小于3B参数的模型通过层复制性能持续提升，而7B以上模型则下降，因此该方法适用于小模型实用化。CircuitProbe仅需10个校准样本，且在英语、印地语、中文和法语下预测结果稳定。
原文摘要 · Abstract (English)
Transformer language models contain localized reasoning circuits, contiguous layer blocks that improve reasoning when duplicated at inference time. Finding these circuits currently requires brute-force sweeps costing 25 GPU hours per model. We propose CircuitProbe, which predicts circuit locations from activation statistics in under 5 minutes on CPU, providing a speedup of three to four orders of magnitude. We find that reasoning circuits come in two types: stability circuits in early layers, detected through the derivative of representation change, and magnitude circuits in late layers, detected through anomaly scoring. We validate across 9 models spanning 6 architectures, including 2025 models, confirming that CircuitProbe top predictions match or are within 2 layers of the optimal circuit in all validated cases. A scaling experiment across the Qwen 2.5 family reveals that layer duplication consistently benefits models under 3B parameters but degrades performance in 7B+ models, making this a practical scaling technique for small language models. CircuitProbe requires as few as 10 calibration examples and its predictions are stable across English, Hindi, Chinese, and French.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。