用隐式欧拉迭代提升Transformer效率与性能，兼顾压缩与推理速度。
IIET: Efficient Numerical Transformer via Implicit Iterative Euler Method
- 采用迭代隐式欧拉法简化高阶微分方程建模，降低计算开销。
- 在lm-evaluation-harness上比基础Transformer提升2.65%准确率，比PCformer高0.8%。
- 支持灵活压缩，最高效版本保留99.4%精度，推理耗时减少55%。
高阶数值方法虽能提升Transformer在自然语言处理与计算机视觉任务中的表现，但带来计算开销增加的效率代价。我们发现，传统压缩技术如知识蒸馏可能损害此类模型性能，以PCformer为例。为此，提出迭代隐式欧拉Transformer（IIET），通过迭代隐式欧拉法简化高阶方法，不仅实现更优性能，还利于模型压缩。为提升推理效率，引入迭代影响感知蒸馏（IIAD），通过可调阈值灵活平衡性能与效率。在lm-evaluation-harness上，IIET相比基础Transformer平均准确率提升2.65%，比PCformer高0.8%。其高效变体E-IIET将推理开销降低55%，同时保持99.4%的原始任务精度。最高效的版本在接近同等速度下，平均性能超越基础Transformer超过1.6%。
原文摘要 · Abstract (English)
High-order numerical methods enhance Transformer performance in tasks like NLP and CV, but introduce a performance-efficiency trade-off due to increased computational overhead. Our analysis reveals that conventional efficiency techniques, such as distillation, can be detrimental to the performance of these models, exemplified by PCformer. To explore more optimizable ODE-based Transformer architectures, we propose the Iterative Implicit Euler Transformer (IIET), which simplifies high-order methods using an iterative implicit Euler approach. This simplification not only leads to superior performance but also facilitates model compression compared to PCformer. To enhance inference efficiency, we introduce Iteration Influence-Aware Distillation (IIAD). Through a flexible threshold, IIAD allows users to effectively balance the performance-efficiency trade-off. On lm-evaluation-harness, IIET boosts average accuracy by 2.65% over vanilla Transformers and 0.8% over PCformer. Its efficient variant, E-IIET, significantly cuts inference overhead by 55% while retaining 99.4% of the original task accuracy. Moreover, the most efficient IIET variant achieves an average performance gain exceeding 1.6% over vanilla Transformer with comparable speed.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。