arXiv:2601.19672cs.LGcs.AI2026-01

为联邦大模型生成文本提供逐令牌溯源,精准定位贡献者。

ProToken: Token-Level Attribution for Federated Large Language Models

  • 基于注意力机制后期层聚焦与梯度加权,实现逐令牌溯源。
  • 在16种配置下平均溯源准确率达98%,支持多客户端扩展。
  • 适用于医疗、金融等高可信场景,助力安全审计与公平激励。

联邦学习(FL)使大语言模型(LLM)能在分布式数据源上协同训练并保护隐私。然而,在关键应用中部署联邦LLM时,仍无法明确特定生成文本由哪些客户端贡献,阻碍了调试、恶意客户端识别、公平奖励分配和信任验证。本文提出ProToken,一种面向联邦大模型的逐令牌溯源方法,在保持联邦学习隐私约束的同时,实现自回归文本生成中的客户端溯源。ProToken利用两个关键洞察:(1)Transformer架构中任务信号集中于后期块,可通过选择性层提升计算效率;(2)基于梯度的相关性加权可过滤无关神经激活,聚焦影响生成结果的关键神经元。我们在四种模型(Gemma、Llama、Qwen、SmolLM)和四个领域(医疗、金融、数学、编程)共16种配置下评估该方法。ProToken在正确识别责任客户端方面达到平均98%的溯源准确率,并在客户端数量增加时仍保持高精度,验证了其在真实场景中的可行性。

原文摘要 · Abstract (English)

Federated Learning (FL) enables collaborative training of Large Language Models (LLMs) across distributed data sources while preserving privacy. However, when federated LLMs are deployed in critical applications, it remains unclear which client(s) contributed to specific generated responses, hindering debugging, malicious client identification, fair reward allocation, and trust verification. We present ProToken, a novel Provenance methodology for Token-level attribution in federated LLMs that addresses client attribution during autoregressive text generation while maintaining FL privacy constraints. ProToken leverages two key insights to enable provenance at each token: (1) transformer architectures concentrate task-specific signals in later blocks, enabling strategic layer selection for computational tractability, and (2) gradient-based relevance weighting filters out irrelevant neural activations, focusing attribution on neurons that directly influence token generation. We evaluate ProToken across 16 configurations spanning four LLM architectures (Gemma, Llama, Qwen, SmolLM) and four domains (medical, financial, mathematical, coding). ProToken achieves 98% average attribution accuracy in correctly localizing responsible client(s), and maintains high accuracy when the number of clients are scaled, validating its practical viability for real-world deployment settings.

联邦学习大模型溯源隐私

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。