让大模型轻松实现长文本高效推理，速度提升1.2倍、内存降1.8倍。
MoDification: Mixture of Depths Made Easy
- 将MoD中的top-k操作改为阈值筛选，降低训练成本
- 在3B到70B模型上实现1.2倍加速与1.8倍内存减少
- 适合希望无痛升级长文本推理能力的工程师
长上下文效率已成为大语言模型服务中的热门话题。混合深度（MoD）被提出作为降低延迟和内存的理想方案。然而，本文发现，若无大量令牌的昂贵训练，现有大模型难以通过MoD实现有效转换。为此，我们提出改进方法：将MoD中的top-k操作升级为阈值选择（threshold-p），并同步优化架构与数据。该系列设计构成新方法「MoDification」。在覆盖3B至70B模型规模的全面实验中，验证其在效率与效果间取得卓越平衡。在长文本应用中，相比原始模型，可实现约1.2倍的延迟加速与约1.8倍的内存减少。
原文摘要 · Abstract (English)
Long-context efficiency has recently become a trending topic in serving large language models (LLMs). And mixture of depths (MoD) is proposed as a perfect fit to bring down both latency and memory. In this paper, however, we discover that MoD can barely transform existing LLMs without costly training over an extensive number of tokens. To enable the transformations from any LLMs to MoD ones, we showcase top-k operator in MoD should be promoted to threshold-p operator, and refinement to architecture and data should also be crafted along. All these designs form our method termed MoDification. Through a comprehensive set of experiments covering model scales from 3B to 70B, we exhibit MoDification strikes an excellent balance between efficiency and effectiveness. MoDification can achieve up to ~1.2x speedup in latency and ~1.8x reduction in memory compared to original LLMs especially in long-context applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。