用动态规划优化非线性函数近似，让大模型推理快4倍
NLI:Non-uniform Linear Interpolation Approximation of Nonlinear Operations for Efficient LLMs Inference
- 通过动态规划自动选最优分段点，精准逼近非线性函数
- 在多个模型上实现接近零精度损失，计算效率提升超4倍
- 无需校准的通用单元，适配各类大模型加速部署
大型语言模型在多项任务中表现卓越，但其部署常受限于庞大的内存占用和计算开销。尽管先前研究已在压缩与加速线性层方面取得显著进展，非线性层（如SiLU、RMSNorm、Softmax）仍严重依赖高精度浮点运算。本文提出一种免校准、基于动态规划最优、硬件友好的框架——非均匀线性插值（NLI）。NLI 能高效近似多种非线性函数，可无缝集成至大模型及其他深度神经网络，几乎无精度损失。NLI 巧妙将分段点选择建模为动态规划问题，利用贝尔曼最优性原理，在 O(M×N²) 时间内实现全局最小插值误差。基于该算法，我们设计并实现了即插即用的通用非线性计算单元。硬件实验表明，NLI 引擎相比当前最佳设计，计算效率提升超过4倍。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have demonstrated remarkable performance across a wide range of tasks, but their deployment is often constrained by substantial memory footprints and computational costs. While prior work has achieved significant progress in compressing and accelerating linear layers, nonlinear layers-such as SiLU, RMSNorm, and Softmax-still heavily depend on high-precision floating-point operations. In this paper, we propose a calibration-free, dynamic-programming-optimal, and hardware-friendly framework called Non-uniform Linear Interpolation (NLI). NLI is capable of efficiently approximating a variety of nonlinear functions, enabling seamless integration into LLMs and other deep neural networks with almost no loss in accuracy. NLI ingeniously recasts cutpoint selection as a dynamic-programming problem, achieving the globally minimal interpolation error in O(MxN2) time via Bellman's optimality principle. Based on the NLI algorithm, we also design and implement a plug-and-play universal nonlinear computation unit. Hardware experiments demonstrate that the NLI Engine achieves more than 4x improvement in computational efficiency compared to the state-of-the-art designs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。