手机端高效私密微调大模型，通信量降90%以上
PAE MobiLLM: Privacy-Aware and Efficient LLM Fine-Tuning on the Mobile Device via Additive Side-Tuning
- 通过加法式旁路网络实现手机端私密微调
- 激活缓存与快捷传输使通信量减少90%以上
- 适合注重隐私和算力受限的移动端应用
现有基于服务器辅助的大语言模型（LLM）微调方法在移动设备上存在通信开销大、数据泄露风险等问题。为此，本文提出PAE MobiLLM，一种隐私感知且高效的移动端LLM微调方法。该方法采用加法式旁路网络设计，服务器仅根据设备定义的预测差异训练适配模块，不接触原始数据和标签。通过在服务端引入激活缓存机制，可复用历史激活，避免设备重复计算；同时设计激活快捷通道，仅传输损失计算相关的token，显著降低通信量。实验表明，该方法在保持高精度的同时，通信成本降低超过90%。
原文摘要 · Abstract (English)
There is a huge gap between numerous intriguing applications fostered by on-device large language model (LLM) fine-tuning (FT) from fresh mobile data and the limited resources of a mobile device. While existing server-assisted methods (e.g., split learning or side-tuning) may enable LLM FT on the local mobile device, they suffer from heavy communication burdens of activation transmissions, and may disclose data and labels to the server. To address those issues, we develop PAE MobiLLM, a a privacy-aware and efficient LLM FT method which can be deployed on the mobile device via server-assisted additive side-tuning. To further accelerate FT convergence and improve computing efficiency, PAE MobiLLM integrates activation caching on the server side, which allows the server to reuse historical activations and saves the mobile device from repeatedly computing forward passes for the recurring data samples. Besides, to reduce communication cost, PAE MobiLLM develops an activation shortcut that transmits only the token involved in the loss calculation instead of full activation matrices to guide the side network tuning. Last but not least, PAE MobiLLM introduces the additive adapter side-network design which makes the server train the adapter modules based on device-defined prediction differences rather than raw ground-truth labels. In this way, the server can only assist device-defined side-network computing, and learn nothing about data and labels. Extensive experimental results demonstrate PAE MobiLLM's superiority.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。