通过错补偿机制提升大模型在低资源设备上的推理效率与精度。
HCInfer: An Efficient Inference System via Error Compensation for Resource-Constrained Devices
- 将补偿分支卸载到CPU，GPU专注执行压缩主干网络。
- 相比压缩模型最高提升5.2%准确率，比全精度模型快10.4倍。
- 适合移动端、边缘设备部署大语言模型的场景。
大型语言模型因参数量巨大，在消费级硬件上部署常受内存限制。现有压缩和卸载方案虽提升可行性，但往往导致精度下降或吞吐瓶颈。近期错补偿方法通过类似LoRA的辅助分支恢复精度，我们发现这些分支虽需大量参数存储，但每次推理仅访问少量补偿参数，具备天然可卸载性。基于此，我们提出HCInfer——一种异构推理系统，将残差补偿卸载至CPU，同时在GPU上运行压缩主干网络，并引入异步补偿流水线与敏感度感知动态秩分配策略，以隐藏补偿开销并最大化精度恢复。实验表明，相较于压缩模型，HCInfer在下游任务中最高提升5.2%准确率；相比全精度模型，最高实现10.4倍加速。
原文摘要 · Abstract (English)
LLMs often struggle with memory-constrained deployment on consumer-grade hardware due to their massive parameter sizes. While existing solutions such as model compression and offloading improve deployment feasibility, they often suffer from substantial accuracy degradation or severe throughput bottlenecks. Recent error compensation methods recover accuracy through auxiliary LoRA-style branches, and we observe that these branches are inherently amenable to offloading: they require substantial parameter storage but access only a small subset of compensation parameters during each inference step. Motivated by this opportunity, we propose HCInfer, a heterogeneous inference system that offloads residual compensation to the CPU while executing the compressed backbone on the GPU, and further introduces an asynchronous compensation pipeline and sensitivity-aware dynamic rank allocation to hide compensation overhead and maximize accuracy recovery. Experimental results show that HCInfer achieves a maximum accuracy improvement of 5.2% on downstream tasks compared to compression model and sustaining a maximum speedup of 10.4x compared to full-precision model.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。