对比神经符号与大模型在农业对话信息抽取中的表现
Information Extraction from Conversation Transcripts: Neuro-Symbolic vs. LLM
- 采用神经符号与大模型两种方法提取农业访谈信息
- 大模型总F1达69.4，显著高于神经符号的52.7
- 适合注重性能的场景，也提醒权衡效率与可控性
当前信息抽取（IE）过度依赖大语言模型，忽视了多年积累的符号化或统计型系统经验。本文在畜牧、乳品和作物三个农业子领域，对神经符号（NS）与基于大模型（LLM）的IE系统进行了比较，评估了九次访谈数据。结果显示，大模型系统整体表现更优（总F1：69.4 vs. 52.7；核心信息F1：63.0 vs. 47.2），其中总信息包括所有抽取内容，核心信息聚焦关键细节。然而两者各有权衡：神经符号方法运行更快、控制更强，在无上下文任务中精度高，但泛化能力差，难以处理语境细节，且开发维护成本高；大模型系统性能更高、部署更快、维护更易，但运行慢、控制弱、依赖模型且存在幻觉风险。研究揭示了实际应用中部署NLP系统的‘隐性成本’，强调需在性能、效率与可控性之间取得平衡。
原文摘要 · Abstract (English)
The current trend in information extraction (IE) is to rely extensively on large language models, effectively discarding decades of experience in building symbolic or statistical IE systems. This paper compares a neuro-symbolic (NS) and an LLM-based IE system in the agricultural domain, evaluating them on nine interviews across pork, dairy, and crop subdomains. The LLM-based system outperforms the NS one (F1 total: 69.4 vs. 52.7; core: 63.0 vs. 47.2), where total includes all extracted information and core focuses on essential details. However, each system has trade-offs: the NS approach offers faster runtime, greater control, and high accuracy in context-free tasks but lacks generalizability, struggles with contextual nuances, and requires significant resources to develop and maintain. The LLM-based system achieves higher performance, faster deployment, and easier maintenance but has slower runtime, limited control, model dependency and hallucination risks. Our findings highlight the "hidden cost" of deploying NLP systems in real-world applications, emphasizing the need to balance performance, efficiency, and control.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。