根据输入动态调整每层精度，让本地大模型运行更快更准。
DP-LLM: Runtime Model Adaptation with Dynamic Layer-wise Precision Assignment
- 按解码过程动态分配各层精度，适应不同运行需求。
- 在多个模型和基准上实现更好性能与延迟平衡。
- 适合资源受限设备上高效部署大语言模型的场景。
如何有效应对本地大语言模型（LLM）在不同运行约束（如延迟、准确率）下的查询？多尺度量化通过叠加不同位宽量化后的多个模型变体，实现了内存高效的运行时模型自适应。然而，一个关键问题仍待解决：如何正确配置模型以匹配目标精度或延迟？尽管混合精度已具潜力，本文进一步提出基于关键观察——各层对精度的敏感性随解码步骤动态变化——设计了DP-LLM机制，依据输入值动态为每层分配精度。在多个模型和基准上的实验结果表明，DP-LLM在性能-延迟权衡上优于现有方法。
原文摘要 · Abstract (English)
How can we effectively handle queries for on-device large language models (LLMs) with varying runtime constraints, such as latency and accuracy? Multi-scale quantization addresses this challenge by enabling memory-efficient runtime model adaptation of LLMs through the overlaying of multiple model variants quantized to different bitwidths. Meanwhile, an important question still remains open-ended: how can models be properly configured to match a target precision or latency? While mixed-precision offers a promising solution, we take this further by leveraging the key observation that the sensitivity of each layer dynamically changes across decoding steps. Building on this insight, we introduce DP-LLM, a novel mechanism that dynamically assigns precision to each layer based on input values. Experimental results across multiple models and benchmarks demonstrate that DP-LLM achieves a superior performance-latency trade-off, outperforming prior approaches.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。