让蒸馏模型支持中途退出，显著提升Transformer推理速度。
LEAP: Layer-wise Exit-Aware Pretraining for Efficient Transformer Inference
- 在蒸馏时加入层对齐约束，使中间层逼近最终层表示。
- 实测推理速度提升1.61倍，91.9%样本在第7层前退出。
- 无需改结构，适合追求高效部署的NLP系统开发者。
层对齐蒸馏与基于收敛性的中途退出是Transformer推理效率的两大主流方法；然而我们发现，在标准部署条件下二者存在系统性不兼容。对齐中间层与教师表示的蒸馏目标会抑制中途退出机制所依赖的表征收敛性，导致蒸馏模型无法有效应用中途退出。为此，我们提出LEAP（层级退出感知预训练），一种无需架构修改的辅助训练目标，通过单一约束确保中间层近似最终层表示。LEAP-MiniLM在batch=1、NVIDIA L4环境下实现1.61×的实测墙钟加速（θ=0.95），91.9%样本在第7层前退出，理论层数减少达1.80×，而标准蒸馏模型有效加速为零。我们在句子相似性（STS-B: 0.760±0.006）和检索基准（BEIR）上验证，提供延迟测量、决策阈值与部署准则等实用指导。
原文摘要 · Abstract (English)
Layer-aligned distillation and convergence-based early exit represent two predominant computational efficiency paradigms for transformer inference; yet we establish that they exhibit systematic incompatibility under standard deployment conditions for convergence-based early exit. Distillation objectives that align intermediate student layers to teacher representations suppress the representational convergence that early-exit mechanisms exploit, rendering such mechanisms ineffective on distilled models. We introduce LEAP (Layer-wise Exit-Aware Pretraining), an auxiliary training objective that reconciles this incompatibility. LEAP requires no architectural modifications; it augments standard distillation with a single constraint ensuring intermediate layers approximate final-layer representations. LEAP-MiniLM achieves 1.61$\times$ measured wall-clock speedup (batch=1, NVIDIA L4) at $θ$=0.95, with 91.9% of samples exiting by layer 7 and 1.80$\times$ theoretical layer reduction, where standard distilled models achieve zero effective speedup. We validate across sentence similarity (STS-B: 0.760 $\pm$ 0.006) and retrieval benchmarks (BEIR), providing operational guidance including latency measurements, decision thresholds, and deployment criteria.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。