解决联邦学习中模型层间差异问题，实现更精准的个性化微调。
FedTreeLoRA: Reconciling Statistical and Functional Heterogeneity in Federated LoRA Fine-Tuning
- 采用树状结构动态聚合，分层实现参数共享与个性化。
- 在NLU和NLG任务上超越现有方法，提升模型泛化与定制能力。
- 适合需要兼顾隐私保护与模型个性化的实际应用场景。
联邦学习结合低秩适配（LoRA）已成为保护隐私的大型语言模型微调标准。然而，现有个性化方法大多基于单一模型假设：仅处理客户端间的统计异构性，将模型视为整体块，忽略了语言模型各层间的功能异构性。我们指出，这两种异构性——横向的统计差异与纵向的功能差异——在源头上正交，但在交互中耦合，表明最优的参数共享深度应依赖于客户端相似度。为此，我们提出FedTreeLoRA框架，采用树状聚合实现细粒度、逐层对齐。通过动态构建聚合层次结构，允许客户端在浅层‘主干’共享共识，同时在深层‘分支’逐步实现专业化。在自然语言理解（NLU）和自然语言生成（NLG）基准上的实验表明，FedTreeLoRA显著优于现有最先进方法，有效协调了通用性与个性化。
原文摘要 · Abstract (English)
Federated Learning (FL) with Low-Rank Adaptation (LoRA) has become a standard for privacy-preserving LLM fine-tuning. However, existing personalized methods predominantly operated under a restrictive Flat-Model Assumption: they addressed client-side \textit{statistical heterogeneity} but treated the model as a monolithic block, ignoring the \textit{functional heterogeneity} across LLM layers. We argue that these two statistical (horizontal) and functional (vertical) dimensions, are \textit{orthogonal in source yet coupled in interaction}, implying that the optimal depth of parameter sharing is functionally dependent on client similarity. To address this, we propose \textbf{FedTreeLoRA}, a framework employing tree-structured aggregation for fine-grained, layer-wise alignment. By dynamically constructing an aggregation hierarchy, FedTreeLoRA allows clients to share broad consensus on shallow `trunks' while progressively specializing on deep `branches'. Experiments on NLU and NLG benchmarks demonstrate that FedTreeLoRA significantly outperforms state-of-the-art methods by effectively reconciling generalization and personalization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。