用双压缩策略大幅降低ViT分层学习的通信开销，保持高精度。
Communication Efficient Split Learning of ViTs with Attention-based Double Compression
- 基于注意力分数合并相似激活，跨类别通用
- 删减意义最小的特征令牌，进一步压缩数据量
- 无需额外调参，自然实现梯度压缩，适合边缘设备
本文提出一种新型通信高效的分层学习框架ADC（Attention-based Double Compression），用于降低视觉变换器（ViTs）在分层学习训练中传输中间激活值的通信开销。ADC采用两种并行压缩策略：第一，基于客户端最后一层的平均注意力得分，合并相似样本的激活，该策略不依赖类别信息，可跨类别合并且不影响模型泛化能力；第二，在此基础上剔除意义最小的特征令牌，进一步降低通信成本。结合两种策略不仅减少了前向传播时的数据传输量，还使梯度天然被压缩，从而无需额外梯度近似或调参即可完成完整模型训练。仿真结果表明，该方法显著降低通信开销，同时保持高精度，优于现有先进分层学习框架。
原文摘要 · Abstract (English)
This paper proposes a novel communication-efficient Split Learning (SL) framework, named Attention-based Double Compression (ADC), which reduces the communication overhead required for transmitting intermediate Vision Transformers activations during the SL training process. ADC incorporates two parallel compression strategies. The first one merges samples' activations that are similar, based on the average attention score calculated in the last client layer; this strategy is class-agnostic, meaning that it can also merge samples having different classes, without losing generalization ability nor decreasing final results. The second strategy follows the first and discards the least meaningful tokens, further reducing the communication cost. Combining these strategies not only allows for sending less during the forward pass, but also the gradients are naturally compressed, allowing the whole model to be trained without additional tuning or approximations of the gradients. Simulation results demonstrate that Attention-based Double Compression outperforms state-of-the-art SL frameworks by significantly reducing communication overheads while maintaining high accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。