arXiv:2604.15764cs.LGcs.AI2026-04

提出首个自适应深度网络的泛化理论,解释为何早退出模型既快又准。

When Do Early-Exit Networks Generalize? A PAC-Bayesian Theory of Adaptive Depth

  • 基于退出深度熵和期望深度构建新泛化界,突破传统最大深度限制。
  • 理论证明自适应深度模型在特定条件下性能优于固定深度模型。
  • 可指导实际部署中退出阈值设置,误差仅0.1-0.3%。

早退出神经网络通过允许高置信度预测在中间层提前输出,实现2-8倍推理加速。尽管广泛应用,其泛化能力缺乏理论支持,此问题已在近期综述中明确指出。本文建立统一的PAC-Bayesian框架,用于自适应深度网络分析:(1)提出新型基于熵的泛化界,依赖退出深度熵H(D)与期望深度E[D],样本复杂度为O((E[D]·d + H(D))/ε²);(2)给出完整推导的显式主系数√(2ln2)≈1.177;(3)建立充分条件,证明自适应深度模型可严格优于固定深度模型;(4)将标签独立假设放宽至ε-近似策略,提升对学习路由机制的适用性;(5)在6个架构、7个基准上的实验表明,所提边界的紧致性比经典界高1.52-3.87倍(全部p<0.001),且边界引导的阈值选择与验证调优性能差距仅0.1-0.3%。

原文摘要 · Abstract (English)

Early-exit neural networks enable adaptive computation by allowing confident predictions to exit at intermediate layers, achieving 2-8$\times$ inference speedup. Despite widespread deployment, their generalization properties lack theoretical understanding -- a gap explicitly identified in recent surveys. This paper establishes a unified PAC-Bayesian framework for adaptive-depth networks. (1) Novel Entropy-Based Bounds: We prove the first generalization bounds depending on exit-depth entropy $H(D)$ and expected depth $\mathbb{E}[D]$ rather than maximum depth $K$, with sample complexity $\mathcal{O}((\mathbb{E}[D] \cdot d + H(D))/ε^2)$. (2) Explicit Constructive Constants: Our analysis yields the leading coefficient $\sqrt{2\ln 2} \approx 1.177$ with complete derivation. (3) Provable Early-Exit Advantages: We establish sufficient conditions under which adaptive-depth networks strictly outperform fixed-depth counterparts. (4) Extension to Approximate Label Independence: We relax the label-independence assumption to $ε$-approximate policies, broadening applicability to learned routing. (5) Comprehensive Validation: Experiments across 6 architectures on 7 benchmarks demonstrate tightness ratios of 1.52-3.87$\times$ (all $p < 0.001$) versus $>$100$\times$ for classical bounds. Bound-guided threshold selection matches validation-tuned performance within 0.1-0.3%.

自适应计算泛化理论早退出机器学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。