arXiv:2601.05143cs.CVcs.CL2026-01

轻量级双阶段模型实现作物病害问答,准确率超99%且可解释。

A Two-Stage Multitask Vision-Language Framework for Explainable Crop Disease Visual Question Answering

  • 分两阶段训练:先多任务预训练视觉编码器,再微调语言解码器
  • 植物与病害识别准确率分别达99.94%和99.06%,生成质量高
  • 支持用户自定义提问,可视化注意力提升结果可信度

作物病害视觉问答需要精准的视觉理解与可靠的文本生成。本文提出一种轻量且可解释的多模态框架,基于Swin Transformer视觉编码器与序列到序列语言解码器,采用两阶段训练策略:先在植物与病害分类任务上联合训练视觉编码器,再冻结其参数并训练文本解码器,增强视觉表征与跨模态对齐。在大规模Crop Disease Domain Multimodal(CDDM)数据集上评估,模型达到99.94%植物分类准确率与99.06%病害分类准确率,并在自然语言生成指标(BLEU、ROUGE、BERTScore)上表现优异。无需微调即可在外部PlantVillageVQA基准上实现83.18%的VQA微平均准确率。模型参数远少于大型基线仍性能更优。通过Grad-CAM与词级归因分析,提供可解释的视觉与文本证据。定性结果显示其在多样用户提问下仍具鲁棒性。交互式演示已通过Hugging Face Gradio公开可用。

原文摘要 · Abstract (English)

Visual question answering (VQA) for crop disease analysis requires accurate visual understanding and reliable language generation. In this work, we present a lightweight and explainable vision-language framework for crop and disease identification from leaf images. The proposed approach integrates a Swin Transformer vision encoder with sequence-to-sequence language decoders. The vision encoder is first trained in a multitask setup for both plant and disease classification, and then frozen while the text decoders are trained, forming a two-stage training strategy that enhances visual representation learning and cross-modal alignment. We evaluate the model on the large-scale Crop Disease Domain Multimodal (CDDM) dataset using both classification and natural language generation metrics. Experimental results demonstrate near-perfect recognition performance, achieving 99.94% plant classification accuracy and 99.06% disease classification accuracy, along with strong BLEU, ROUGE and BERTScore results. Without fine-tuning, the model further generalizes well to the external PlantVillageVQA benchmark, achieving 83.18% micro accuracy in the VQA task. Our lightweight design outperforms larger vision-language baselines while using significantly fewer parameters. Explainability is assessed through Grad-CAM and token-level attribution, providing interpretable visual and textual evidence for predictions. Qualitative results demonstrate robust performance under diverse user-driven queries, highlighting the effectiveness of task-specific visual pretraining and the two-stage training methodology for crop disease visual question answering. An interactive demo of the proposed Swin-T5 model is publicly available as a Gradio-based application at https://huggingface.co/spaces/Zahid16/PlantDiseaseVQAwithSwinT5 for community use.

作物病害视觉问答可解释性轻量模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。