对比6种工具识别模糊实体，发现LLM更懂语境但传统工具更稳。
Is 'Hope' a person or an idea? A pilot benchmark for NER: comparing traditional NLP tools and large language models on ambiguous entities
- 用人工标注119个词的基准测试，评估6种系统在5类实体上的表现
- LLM平均F1更高，Gemini在人名识别上最优，传统工具在日期位置更一致
- 适合关注实体识别选型或语言模型局限性的研究者参考
本小规模试点研究构建了一个精心标注的命名实体识别（NER）基准数据集，涵盖119个标记，涉及五类实体（PERSON、LOCATION、ORGANIZATION、DATE、TIME）。评估了三种非大语言模型（NLTK、spaCy、Stanza）与三种通用大语言模型（Gemini-1.5-flash、DeepSeek-V3、Qwen-3-4B）的表现，使用F1分数与人工标注标准对比。结果显示，大语言模型在语境敏感实体（如人名）识别上普遍优于传统工具，其中Gemini达到最高平均F1；而Stanza等传统工具在结构化标签（如地点、日期）上表现出更强一致性。同时观察到不同大模型在时间表达和多词组织识别上存在差异。研究表明，尽管大语言模型具备更强上下文理解能力，传统工具在特定任务中仍具竞争力，为模型选择提供依据。
原文摘要 · Abstract (English)
This pilot study presents a small-scale but carefully annotated benchmark of Named Entity Recognition (NER) performance across six systems: three non-LLM NLP tools (NLTK, spaCy, Stanza) and three general-purpose large language models (LLMs: Gemini-1.5-flash, DeepSeek-V3, Qwen-3-4B). The dataset contains 119 tokens covering five entity types (PERSON, LOCATION, ORGANIZATION, DATE, TIME). We evaluated each system's output against the manually annotated gold standard dataset using F1-score. The results show that LLMs generally outperform conventional tools in recognizing context-sensitive entities like person names, with Gemini achieving the highest average F1-score. However, traditional systems like Stanza demonstrate greater consistency in structured tags such as LOCATION and DATE. We also observed variability among LLMs, particularly in handling temporal expressions and multi-word organizations. Our findings highlight that while LLMs offer improved contextual understanding, traditional tools remain competitive in specific tasks, informing model selection.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。