将敏感数据本地处理,非敏感数据远程协同,实现隐私与性能的平衡。
PWC-MoE: Privacy-Aware Wireless Collaborative Mixture of Experts
- 按数据敏感性动态分配至本地或远程专家,仅一个专家处理每条数据。
- 在低带宽下仍保持高模型性能,敏感数据本地处理率达90%以上。
- 适合医疗、金融等隐私敏感且网络受限的场景使用。
大型语言模型(LLMs)部署在云端可减轻本地设备的计算与存储负担,但因敏感数据传输引发隐私担忧,并需大量通信带宽,在资源受限环境中面临挑战。相比之下,本地运行的小型语言模型(SLMs)虽提升隐私保护,却在复杂任务上表现有限。为在带宽受限条件下平衡计算成本、性能与隐私保护,本文提出隐私感知的无线协同专家混合模型(PWC-MoE)。该框架采用稀疏隐私感知门控网络,将敏感令牌动态路由至本地客户端的隐私专家,非敏感令牌则路由至远程基站的非隐私专家。门控网络确保每个令牌仅由一个专家处理,以提升效率。为增强可扩展性并防止特定专家过载,引入分组负载均衡机制,均匀分配敏感与非敏感令牌。为适应带宽限制同时保持模型性能,提出带宽自适应且重要性感知的令牌卸载方案,通过重要性预测器评估非敏感令牌的重要性,基于预测值和可用带宽优先传输关键令牌。实验表明,该框架在带宽受限环境下有效保护隐私并维持高性能,为隐私敏感且带宽受限场景下的LLM部署提供可行解决方案。
原文摘要 · Abstract (English)
Large language models (LLMs) hosted on cloud servers alleviate the computational and storage burdens on local devices but raise privacy concerns due to sensitive data transmission and require substantial communication bandwidth, which is challenging in constrained environments. In contrast, small language models (SLMs) running locally enhance privacy but suffer from limited performance on complex tasks. To balance computational cost, performance, and privacy protection under bandwidth constraints, we propose a privacy-aware wireless collaborative mixture of experts (PWC-MoE) framework. Specifically, PWC-MoE employs a sparse privacy-aware gating network to dynamically route sensitive tokens to privacy experts located on local clients, while non-sensitive tokens are routed to non-privacy experts located at the remote base station. To achieve computational efficiency, the gating network ensures that each token is dynamically routed to and processed by only one expert. To enhance scalability and prevent overloading of specific experts, we introduce a group-wise load-balancing mechanism for the gating network that evenly distributes sensitive tokens among privacy experts and non-sensitive tokens among non-privacy experts. To adapt to bandwidth constraints while preserving model performance, we propose a bandwidth-adaptive and importance-aware token offloading scheme. This scheme incorporates an importance predictor to evaluate the importance scores of non-sensitive tokens, prioritizing the most important tokens for transmission to the base station based on their predicted importance and the available bandwidth. Experiments demonstrate that the PWC-MoE framework effectively preserves privacy and maintains high performance even in bandwidth-constrained environments, offering a practical solution for deploying LLMs in privacy-sensitive and bandwidth-limited scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。