通过手工构造语言学例句，诊断并预测序列标注模型在真实场景的表现。
Towards a Diagnostic and Predictive Evaluation Methodology for Sequence Labeling Tasks
- 基于语言学属性手工构建小规模测试集，覆盖常见标注模式。
- 预测外部数据表现的中位相关性达0.85，优于传统评估。
- 适合需要精准定位模型缺陷与选型的开发者和研究者。
自然语言处理中的标准评估通常仅表明系统A平均优于系统B，但无法提供改进方向，且系统B在外部数据上反而可能更优。本文提出一种面向序列标注任务的诊断性与预测性评估方法，其核心是基于错误分析，通过手工设计少量具有语言学动机的示例构成测试集，全面覆盖实际应用中可能遇到的跨度属性（如形状、长度、大小写、句中位置等），而非依赖大规模真实分布数据收集。该方法在西班牙语英语词识别基准上验证，结果具有诊断性（可识别系统性弱点）、可操作性（指导模型选型）和预测性：对新数据集的表现预测中位相关性达0.85。
原文摘要 · Abstract (English)
Standard evaluation in NLP typically indicates that system A is better on average than system B, but it provides little info on how to improve performance and, what is worse, it should not come as a surprise if B ends up being better than A on outside data. We propose an evaluation methodology for sequence labeling tasks grounded on error analysis that provides both quantitative and qualitative information on where systems must be improved and predicts how models will perform on a different distribution. The key is to create test sets that, contrary to common practice, do not rely on gathering large amounts of real-world in-distribution scraped data, but consists in handcrafting a small set of linguistically motivated examples that exhaustively cover the range of span attributes (such as shape, length, casing, sentence position, etc.) a system may encounter in the wild. We demonstrate this methodology on a benchmark for anglicism identification in Spanish. Our methodology provides results that are diagnostic (because they help identify systematic weaknesses in performance), actionable (because they can inform which model is better suited for a given scenario) and predictive: our method predicts model performance on external datasets with a median correlation of 0.85.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。