用因果干预评估注意力层重要性，实现高效推理
CausalGate: Causal Importance Distillation for Transformer Module Pruning
- 通过干预隔离各模块，测量语义损失以评估重要性
- 在多个模型上实现更低延迟，优于现有动态路由方法
- 生成静态门控参数，无运行时开销，适合部署
现有大语言模型自适应推理方法依赖观测启发式指标（如隐藏状态相似性或激活幅度）来跳过冗余模块。然而，这些基于相关性的度量常无法捕捉对语义准确至关重要的细微非线性结构计算。我们提出CausalGate，一种面向计算高效的Transformer推理的干预引导框架。校准阶段，CausalGate隔离每个Attention和MLP子层，将其输出置零，并通过最终logit分布的Kullback-Leibler散度精确测量语义损伤。为消除运行时路由开销，该结构重要性层级被提炼为一组全局静态轻量标量门控，使用指数移动平均平滑目标与可微成对排序损失实现。在TinyLlama-1.1B、Qwen2.5-3B和Llama-3.1-8B上，跨语言建模与常识推理基准测试中，CausalGate持续优于主流动态路由与层跳过基线，将理论计算节省转化为实际硬件延迟降低，且无操作开销。
原文摘要 · Abstract (English)
Existing adaptive inference methods for Large Language Models rely on observational heuristics, such as hidden-state similarity or activation magnitudes, to drop redundant modules. However, these correlation-based metrics often fail to capture subtle, non-linear structural computations vital for semantic accuracy. We introduce CausalGate, an intervention-guided framework for compute-efficient transformer inference. During a calibration phase, CausalGate isolates individual Attention and MLP sub-layers, zeros out their respective outputs, and measures the exact semantic damage via the Kullback-Leibler divergence of the final logit distribution. To eliminate runtime routing overhead, this structural importance hierarchy is distilled into a global set of static, lightweight scalar gates using an Exponential Moving Average smoothing objective paired with a differentiable pairwise ranking loss. Evaluated on TinyLlama-1.1B, Qwen2.5-3B, and Llama-3.1-8B across language modeling and commonsense reasoning benchmarks, CausalGate consistently outperforms prominent dynamic routing and layer-skipping baselines, translating theoretical compute savings into concrete hardware latency reductions with zero operational overhead.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。