用大模型教小模型识别茶叶病,精度提升且部署更轻便。
Cross-Architecture Knowledge Distillation from a Vision Foundation Model to a Lightweight Visual State Space Model for Tea Leaf Disease Classification

- 从大模型迁移知识到轻量状态空间模型,突破架构差异限制。
- 学生模型参数少5倍,准确率保留98.3%,最高达96.20%。
- 适合资源受限的农业边缘设备,可直接用于田间部署。
自动化茶叶病分类助力精准农业,但受限于计算资源,在边缘设备上部署高精度模型仍具挑战。自监督视觉基础模型如DINOv2具备强大特征,但体积过大难以部署;而从零训练的小型农业模型常因数据不足导致欠拟合。本文研究从微调后的DINOv2(视觉变换器)教师模型向紧凑的双向视觉状态空间模型(LVSSM)学生模型进行跨架构知识蒸馏,该方向尚未被充分探索,因两者使用根本不同的令牌混合机制。我们识别并修复了学生模型在小数据下训练不稳定的两个问题:单一的大卷积嵌入层与切断残差路径的融合层。通过渐进式卷积茎和门控双向选择扫描块,4.45M参数的学生模型实现稳定训练。在三个种子下,温度缩放的逻辑值蒸馏将测试准确率从92.32±2.14%提升至95.41±1.17%(最佳单次运行:96.20%;宏F1:94.45%),平均提升3.09个百分点。学生模型参数仅为2200万教师模型的5.0倍,同时保持98.3%的准确率。消融实验表明,中间特征对齐损失会降低性能,因此仅逻辑值蒸馏效果最佳。公平对比显示,该增益仅出现在初始性能低于教师的学生中。报告了每类指标、混淆矩阵、自助置信区间及FLOPs/延迟测量,并讨论局限性,包括单一数据集范围和简化非官方的SSM实现。
原文摘要 · Abstract (English)
Automated tea leaf disease classification supports precision agriculture, yet deploying accurate models on edge devices remains challenging under tight compute budgets. Self-supervised vision foundation models such as DINOv2 provide strong features but are too large for field deployment, while lightweight models trained from scratch on small agricultural datasets often underfit. We study cross-architecture knowledge distillation (KD) from a fine-tuned DINOv2 teacher (Vision Transformer) to a compact bidirectional Visual State Space Model (LVSSM) student, an underexplored direction because the architectures use fundamentally different token-mixing mechanisms. We identify and fix two training-stability problems that prevent the from-scratch SSM student from learning on limited data: a single large patch-embedding convolution and a fusion layer that severs the residual path. With a progressive convolutional stem and gated bidirectional selective-scan block, the 4.45M-parameter student trains stably. Across three seeds, temperature-scaled logit distillation raises test accuracy from 92.32+/-2.14% to 95.41+/-1.17% (best single run: 96.20%; macro-F1: 94.45%), a +3.09 percentage-point mean gain. The student uses 5.0 times fewer parameters than the 22M-parameter teacher while retaining 98.3% of its accuracy. Ablations show that intermediate feature-alignment losses reduce accuracy, making simple logit-level KD the strongest configuration. A fair from-scratch comparison shows the gain is specific to students that start below the teacher. We report per-class metrics, confusion matrices, bootstrap confidence intervals, and FLOPs/latency measurements, and discuss limitations including the single-dataset scope and simplified non-official SSM implementation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。