跨组织语言模型适配中，仅传输聚合数据，实现可审计的隐私保护训练。
Echelon: Auditable Aggregate-Only Language-Model Adaptation Across Privacy Boundaries
- 设备端本地训练，仅上传安全聚合后的边界差分数据
- 在跨网络延迟、数据异构等条件下保持训练稳定，验证损失3.887
- 适合对数据隐私和审计要求高的企业级模型微调场景
跨组织语言模型适配面临严格的治理约束：设备级模型参数、激活值、优化器状态及单设备更新无法跨管理边界导出。现有分布式与联邦学习框架通常先假设跨站点模型交换，再叠加隐私机制，导致合规复杂且审计脆弱。本文提出Echelon，一种以边界为先的训练架构，将设备级模型状态不可导出作为系统不变量。各边界内设备本地训练，跨边界仅传输经安全聚合的边界级增量与O(1)协调元数据，并通过明确的审计接口暴露。限制交换内容为聚合数据改变了优化问题：系统需在广域网延迟、参与度异质、设备频繁退出及非独立同分布（non-IID）数据条件下保持稳定，而全局层面从未见过单设备更新。Echelon结合缓冲式半异步安全聚合、过时感知加权、参与窗口、近端局部目标及漂移感知外同步控制器。在10亿参数LoRA适配中，跨M=2个边界，三组种子（2488万词元）预算匹配对比下，验证损失达3.887±0.010，在固定词元、固定字节、固定墙钟时间与固定同步次数预算下，表现最优或并列最优。在OpenWebText压力测试中，Echelon维持2139–2176词元/秒吞吐；相比隐私对等的DiLoCo+SA基线，Echelon-DA在高网络延迟下更早达到目标性能；在200毫秒模拟延迟或严重非独立同分布分区下，性能下降不超过2.2%。
原文摘要 · Abstract (English)
Cross-organization language-model adaptation increasingly faces hard governance constraints: in many deployments, device-level model state-parameters, activations, optimizer state, and per-device updates-cannot be exported outside an administrative boundary. Existing distributed and federated stacks typically assume cross-site model exchange and then retrofit privacy mechanisms, which complicates compliance and makes auditing brittle. We present Echelon, a boundary-first training architecture that enforces device-level model-state non-export as a systems invariant. Devices train locally inside each boundary; the only cross-boundary payloads are securely aggregated boundary-level deltas plus O(1) coordination metadata, exposed through a concrete audit surface. Restricting exchange to aggregates changes the optimization problem: the system must remain stable under WAN delay, heterogeneous participation, churn, and non-IID data even though the global plane never sees per-device updates. Echelon combines buffered semi-asynchronous secure aggregation, staleness-aware weighting, participation windows, proximal local objectives, and a drift-aware outer synchronization controller. In 1B-parameter LoRA adaptation across M= 2 boundaries, a budget-matched contest over three seeds (24.88M tokens) reaches validation loss 3.887 +/-0.010 and is best or tied-best among tuned low-communication baselines under fixed-token, fixed-bytes, fixed-wall-clock, and fixed-sync-count budgets. In OpenWebText stress tests, Echelon sustains 2,139-2,176 tokens/s across evaluated WAN and non-IID treatments, Echelon-DA improves time-to-target under WAN latency relative to a privacy-parityDiLoCo+SA baseline, and quality degrades by at most 2.2% under 200ms emulated latency or severe non-IID partitioning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。