让大模型在手机等设备上实时跑医疗问答
Enabling On-Device Medical AI Assistants via Input-Driven Saliency Adaptation
- 根据医疗数据动态剪裁无用神经元，大幅压缩模型
- 压缩后模型在50%和67%大小下仍保持高准确率
- 可在18.7瓦的嵌入式设备上实现低功耗实时推理
大型语言模型（LLMs）在医疗场景中影响深远，但其庞大的体积难以部署于边缘设备等资源受限环境。本文提出一种通用压缩框架，通过在特定领域数据上测量神经元重要性，精准剪除无关神经元，显著减小模型规模并维持性能。剪枝后进一步采用后训练量化降低内存占用，并在MedMCQA、MedQA和PubMedQA等医疗基准测试中评估效果。我们将压缩后的Gemma模型（压缩50%）和LLaMA3模型（压缩67%）部署至Jetson Orin Nano（峰值功耗18.7W）与Raspberry Pi 5（峰值功耗6.3W），实现了在硬件约束下的实时、节能推理。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have significant impact on the healthcare scenarios but remain prohibitively large for deployment in real-time, resource-constrained environments such as edge devices. In this work, we introduce a novel medical assistant system, optimized through our general-purpose compression framework, which tailors Large Language Models (LLMs) for deployment in specialized domains. By measuring neuron saliency on domain-specific data, our method can aggressively prune irrelevant neurons, reducing model size while preserving performance. Following pruning, we apply post-training quantization to further reduce the memory footprint, and evaluate the compressed model across medical benchmarks including MedMCQA, MedQA, and PubMedQA. We also deploy the 50\% compressed Gemma and the 67\% compressed LLaMA3 models on Jetson Orin Nano (18.7W peak) and Raspberry Pi 5 (6.3W peak), achieving real-time, energy-efficient inference under hardware constraints.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。