构建首个用于人道排雷领域的知识提取数据集与评测框架
TextMineX: Data, Evaluation Framework and Ontology-guided LLM Pipeline for Humanitarian Mine Action
- 基于本体引导的LLM流水线，从非结构化报告中提取三元组知识
- 使用柬埔寨排雷中心真实数据，提升模型在真实场景下的表现
- 引入抗偏差评测机制，显著降低幻觉并提高格式一致性
人道排雷（HMA）致力于识别和清除冲突地区地雷，但大量关键操作知识散落在非结构化报告中，阻碍信息共享。为此，我们提出TextMineX：首个面向HMA领域的数据集、评测框架与本体引导的大型语言模型（LLM）知识提取流水线。TextMineX将HMA报告转化为（主体，关系，客体）三元组，构建领域专用知识图谱。为确保现实相关性，数据源自合作方柬埔寨排雷中心（CMAC）。我们进一步设计了一种偏见感知的评测框架，结合人工标注三元组与LLM作为评判者协议，缓解无参考评分中的位置偏差问题。实验表明，本体对齐提示可使抽取准确率提升44.2%，幻觉减少22.5%，格式遵循度提高20.9%。相关数据集与代码已公开发布。
原文摘要 · Abstract (English)
Humanitarian Mine Action (HMA) addresses the challenge of detecting and removing landmines from conflict regions. Much of the life-saving operational knowledge produced by HMA agencies is buried in unstructured reports, limiting the transferability of information between agencies. To address this issue, we propose TextMineX: the first dataset, evaluation framework and ontology-guided large language model (LLM) pipeline for knowledge extraction from text in the HMA domain. TextMineX structures HMA reports into (subject, relation, object)-triples, thus creating domain-specific knowledge. To ensure real-world relevance, we utilized the dataset from our collaborator Cambodian Mine Action Centre (CMAC). We further introduce a bias-aware evaluation framework that combines human-annotated triples with an LLM-as-Judge protocol to mitigate position bias in reference-free scoring. Our experiments show that ontology-aligned prompts improve extraction accuracy by up to 44.2%, reduce hallucinations by 22.5%, and enhance format adherence by 20.9% compared to baseline models. We publicly release the dataset and code.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。