根据输入提示动态剪枝，让大模型更快更准地推理。
Prompt-based Depth Pruning of Large Language Models
- 用轻量路由器根据提示决定跳过哪些模型层
- 在常识推理任务上提速明显，且比固定剪枝更准
- 适合需要快速响应的实时对话系统
深度剪枝通过移除部分不重要的Transformer层来降低大语言模型的推理成本，无需依赖特定硬件。然而我们发现，某一层的重要性高度依赖任务——对某一任务关键的层，在另一任务中可安全移除。基于此，我们提出动态深度剪枝算法PuDDing（Prompt-routed Dynamic Depth Pruning），根据输入提示决定跳过哪些层。PuDDing训练一个轻量级路由器，从数据驱动构建的候选剪枝集合中选择最优组合。在常识推理基准上的实验表明，PuDDing有效加速了模型推理，且在任务表现上优于静态剪枝基线。
原文摘要 · Abstract (English)
Depth pruning aims to reduce the inference cost of a large language model without any hardware-specific complications, by simply removing several less important transformer blocks. However, our empirical findings suggest that the importance of a transformer block may be highly task-dependent -- a block that is crucial for a task can be removed without degrading the accuracy on another task. Based on this observation, we develop a dynamic depth pruning algorithm, coined PuDDing (Prompt-routed Dynamic Depth Pruning), which determines which blocks to omit from the model based on the input prompt. PuDDing operates by training a lightweight router to predict the best omission set among a set of options, where this option set has also been constructed in a data-driven manner. Empirical results on commonsense reasoning benchmarks demonstrate that PuDDing effectively accelerates the inference language models, and achieves better on-task performance than static depth pruning baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。