用神经符号框架把田间笔记和图像转成结构化知识图谱,助力小麦性状精准分析。
PhenoNEST: A Neuro-Symbolic Framework for Ontology-Aware Multimodal Plant Phenotyping and Trait Discovery

- 将田间笔记和图像通过语义对齐构建分层知识图谱
- 在500个样本上实现92.3%的指认准确率和87.1%的视觉词义消歧率
- 适合育种者做田间观测审计与时空性状定位
高通量植物表型技术产生大量数据,常困于非结构化文本和孤立的RGB图像中。为弥合这一语义鸿沟,我们提出一种多模态细粒度知识图谱(KG)构建框架,以追踪基因型-表型随时间与实验的变化关系。研究以小麦(Triticum aestivum)为代表作物,在复杂冠层环境下验证方法有效性。管道首先从嘈杂的田间笔记中提取实体与关系,通过RDF类型化将唯一实例动态转化为层次化类实体,构建初始知识图谱;再利用PlantDeBERTa将图谱节点对齐标准本体(PO、RO、WTO)。为实现视觉锚定,采用视觉-语言模型结合小麦分割ViT生成基于注意力的软图,将特定图谱实体直接关联至图像像素。引入中心观察节点Plant_Obs_Id,按时间连接各多模态子图。在500个精标威斯小麦(WisWheat)样本上,通过指认游戏准确率、视觉词义消歧(VWSD)及基于排名的指标评估,该神经符号方法成功将复杂田间观测映射至结构化图谱,支持自动化田间笔记审计、时序胁迫监测与精确空间性状定位,服务于小麦育种。
原文摘要 · Abstract (English)
High-throughput plant phenotyping generates valuable data that often remains trapped in unstructured text and isolated RGB images. To bridge this semantic gap, we propose a framework for constructing a multimodal granular Knowledge Graph (KG) to monitor genotype-phenotype interactions across time and experiments. In this work, we focus on wheat Triticum aestivum as a representative target crop to validate our methodology across complex canopy environments. Our pipeline first distills noisy field notes to extract entities and relations, dynamically constructing the KG by converting unique instances into hierarchical class entities via RDF-typing. These graph nodes are then aligned with standardized ontologies (PO, RO, WTO) using PlantDeBERTa. To visually ground the constructed graph, a Vision-Language Model paired with a wheat-segmentation ViT generates attention-based softmaps, linking specific KG entities directly to image pixels. We introduce a central observation node Plant_Obs_Id to connect these multimodal subgraphs temporally. Evaluated on 500 curated WisWheat samples using Pointing Game accuracy, Visual Word Sense Disambiguation (VWSD), and rank-based metrics, our neuro-symbolic approach successfully maps complex field observations to a structured graph. This enables automated field note auditing, temporal stress monitoring, and precise spatial trait localization for wheat breeders.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。