为心理健康领域设计隐私保护的联邦大模型微调框架，兼顾安全与性能。
FedMentor: Domain-Aware Differential Privacy for Heterogeneous Federated LLMs in Mental Health
- 按领域定制差分隐私噪声，客户端自适应调整以平衡隐私与效用。
- 在三个心理医疗数据集上，安全输出率提升3个百分点，毒性降低。
- 支持1.7B参数模型，单轮通信量低于173MB，适合资源受限场景。
在敏感领域（如心理健康）中，保护大型语言模型（LLMs）的隐私适应需在严格保密与模型效用、安全性之间取得平衡。本文提出FedMentor，一种融合低秩适配（LoRA）与领域感知差分隐私（DP）的联邦微调框架，使每个客户端（领域）可按数据敏感度自定义DP噪声尺度，服务器在效用下降时动态减少噪声。在三个心理健康数据集上的实验表明，相比无隐私保护的标准联邦学习，FedMentor将安全输出率提升最高达3个百分点，降低内容毒性，同时保持效用（BERTScore F1和ROUGE-L）仅比非私有基线低0.5%，接近集中式上限。该框架可在单GPU客户端上支持高达1.7B参数的模型，每轮通信量不足173MB，为医疗等敏感领域提供可落地的隐私保护微调方案。
原文摘要 · Abstract (English)
Privacy-preserving adaptation of Large Language Models (LLMs) in sensitive domains (e.g., mental health) requires balancing strict confidentiality with model utility and safety. We propose FedMentor, a federated fine-tuning framework that integrates Low-Rank Adaptation (LoRA) and domain-aware Differential Privacy (DP) to meet per-domain privacy budgets while maintaining performance. Each client (domain) applies a custom DP noise scale proportional to its data sensitivity, and the server adaptively reduces noise when utility falls below a threshold. In experiments on three mental health datasets, we show that FedMentor improves safety over standard Federated Learning (FL) without privacy, raising safe output rates by up to three points and lowering toxicity, while maintaining utility (BERTScore F1 and ROUGE-L) within 0.5% of the non-private baseline and close to the centralized upper bound. The framework scales to backbones with up to 1.7B parameters on single-GPU clients, requiring < 173 MB of communication per-round. FedMentor demonstrates a practical approach to privately fine-tune LLMs for safer deployments in healthcare and other sensitive fields.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。