提升分布式卷积计算的抗慢节点能力与数值稳定性
Flexible Coded Distributed Convolution Computing for Enhanced Straggler Resilience and Numerical Stability in Distributed CNNs
- 用编码技术将卷积运算分解为可并行的任务
- 实测在多种卷积网络中提升计算效率与抗延迟能力
- 适合资源受限设备上部署深度学习模型的开发者
在资源受限设备上部署卷积神经网络（CNN）需高效管理计算资源，常通过易受慢节点延迟影响的分布式环境实现。本文提出柔性编码分布式卷积计算（FCDCC）框架，增强分布式CNN中的抗慢节点能力与数值稳定性。将原本用于矩阵乘法的循环与旋转矩阵嵌入（CRME）编码方法扩展至高维张量卷积。针对所提方案——数值稳定编码张量卷积（NSCTC），提出两种新编码分块策略：输入张量的自适应补零编码分块（APCP）与滤波器张量的核通道编码分块（KCCP）。这些策略实现张量卷积的线性分解，并将其编码为编码分布式计算（CDC）子任务，结合模型并行与编码冗余以实现鲁棒高效的执行。理论分析揭示了通信与存储成本间的最优权衡。实证结果验证了该框架在计算效率、抗慢节点能力及可扩展性方面的有效性，适用于多种卷积网络架构。
原文摘要 · Abstract (English)
Deploying Convolutional Neural Networks (CNNs) on resource-constrained devices necessitates efficient management of computational resources, often via distributed environments susceptible to latency from straggler nodes. This paper introduces the Flexible Coded Distributed Convolution Computing (FCDCC) framework to enhance straggler resilience and numerical stability in distributed CNNs. We extend Coded Distributed Computing (CDC) with Circulant and Rotation Matrix Embedding (CRME) which was originally proposed for matrix multiplication to high-dimensional tensor convolution. For the proposed scheme, referred to as the Numerically Stable Coded Tensor Convolution (NSCTC) scheme, we also propose two new coded partitioning schemes: Adaptive-Padding Coded Partitioning (APCP) for the input tensor and Kernel-Channel Coded Partitioning (KCCP) for the filter tensor. These strategies enable linear decomposition of tensor convolutions and encoding them into CDC subtasks, combining model parallelism with coded redundancy for robust and efficient execution. Theoretical analysis identifies an optimal trade-off between communication and storage costs. Empirical results validate the framework's effectiveness in computational efficiency, straggler resilience, and scalability across various CNN architectures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。