让多模态模型在边缘设备高效微调，不传标签、不同步客户端。
Fine-tuning Multimodal Transformers on Edge: A Parallel Split Learning Approach
- 将模型分片并行处理，客户端只保留轻量编码器和分词器。
- 计算量减少250倍，通信开销随模型增大仍保持低水平。
- 适合资源受限场景，尤其适用于多模态任务的分布式训练。
多模态变换器融合图像、音频、文本等多种数据类型，推动了音视频理解与图文检索等任务的发展；然而其高参数量限制了在资源受限边缘设备上的部署。分割学习（SL）通过在特定切分层将模型分块，将计算密集型操作卸载至服务器，为多模态变换器的分布式训练提供了可行路径，但应用仍不充分。本文提出MPSL，一种并行分割学习方法，实现多模态变换器在分布式环境下的高效微调，同时避免标签共享、客户端同步及每客户端子模型管理。MPSL采用轻量级客户端分词器和统一的模态无关编码器，可灵活适配特定任务需求。在7个多模态数据集上的评估表明，MPSL性能达到或优于联邦学习，客户端计算量降低250倍，并在模型规模增长时展现出更优的通信可扩展性。通过深入分析，我们揭示了任务适用性、权衡关系及MPSL表现突出的场景，为后续研究提供启示。
原文摘要 · Abstract (English)
Multimodal transformers integrate diverse data types like images, audio, and text, advancing tasks such as audio-visual understanding and image-text retrieval; yet their high parameterization limits deployment on resource-constrained edge devices. Split Learning (SL), which partitions models at a designated cut-layer to offload compute-intensive operations to the server, offers a promising approach for distributed training of multimodal transformers, though its application remains underexplored. We present MPSL, a parallel SL approach for computational efficient fine-tuning of multimodal transformers in a distributed manner, while eliminating label sharing, client synchronization, and per-client sub-model management. MPSL employs lightweight client-side tokenizers and a unified modality-agnostic encoder, allowing flexible adaptation to task-specific needs. Our evaluation across 7 multimodal datasets demonstrates that MPSL matches or outperforms Federated Learning, reduces client-side computations by 250x, and achieves superior scalability in communication cost with model growth. Through extensive analysis, we highlight task suitability, trade-offs, and scenarios where MPSL excels, inspiring further exploration.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。