用预训练模型特征提升农业图像分类,保护隐私且通信少。
FedReplay: A Feature Replay Assisted Federated Transfer Learning Framework for Efficient and Privacy-Preserving Smart Agriculture
- 用冻结的CLIP视觉模型提取特征,只在客户端训练轻量分类器。
- 共享1%的特征数据使准确率达86.6%,比基线高4倍以上。
- 适合需要隐私保护与低通信开销的智能农业场景。
精准分类在智慧农业中至关重要,可用于作物监测、果实识别和病虫害检测。然而,传统集中式训练需大量数据收集,引发隐私问题;标准联邦学习则面临非独立同分布(non-IID)数据和高通信成本挑战。为此,我们提出一种融合冻结对比语言-图像预训练(CLIP)视觉变压器(ViT)与轻量级变压器分类器的联邦学习框架。通过利用预训练CLIP ViT强大的特征提取能力,避免从头训练大模型,并将联邦更新限制在小型分类器,显著降低传输开销。此外,为缓解非IID数据导致的性能下降,各客户端间共享所有类别中1%的CLIP提取特征表示。这些共享特征无法逆向还原原始图像,保障隐私的同时对齐类表示。在农业分类任务上的实验表明,该方法达到86.6%的准确率,比基线联邦学习高出4倍以上,验证了结合视觉-语言模型特征与联邦学习在隐私保护和可扩展性方面的有效性。
原文摘要 · Abstract (English)
Accurate classification plays a pivotal role in smart agriculture, enabling applications such as crop monitoring, fruit recognition, and pest detection. However, conventional centralized training often requires large-scale data collection, which raises privacy concerns, while standard federated learning struggles with non-independent and identically distributed (non-IID) data and incurs high communication costs. To address these challenges, we propose a federated learning framework that integrates a frozen Contrastive Language-Image Pre-training (CLIP) vision transformer (ViT) with a lightweight transformer classifier. By leveraging the strong feature extraction capability of the pre-trained CLIP ViT, the framework avoids training large-scale models from scratch and restricts federated updates to a compact classifier, thereby reducing transmission overhead significantly. Furthermore, to mitigate performance degradation caused by non-IID data distribution, a small subset (1%) of CLIP-extracted feature representations from all classes is shared across clients. These shared features are non-reversible to raw images, ensuring privacy preservation while aligning class representation across participants. Experimental results on agricultural classification tasks show that the proposed method achieve 86.6% accuracy, which is more than 4 times higher compared to baseline federated learning approaches. This demonstrates the effectiveness and efficiency of combining vision-language model features with federated learning for privacy-preserving and scalable agricultural intelligence.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。