无需标签即可评估分类体系质量，提升评估可靠性。
Reference-Free Evaluation of Taxonomies
- 通过语义与分类相似性相关性衡量分类体系鲁棒性。
- 利用自然语言推理检测分类逻辑是否合理，相关性优于现有方法。
- 可预测下游分类任务表现，适合构建和优化知识体系的团队使用。
我们提出两种无需参考标签的分类体系质量评估指标。首个指标通过计算语义相似性与分类相似性之间的相关性，评估分类体系的鲁棒性，解决了现有指标未覆盖的错误类型。第二个指标利用自然语言推理（Natural Language Inference）评估分类逻辑的合理性。两个指标在五个分类体系上测试，与真实标签的F1分数具有良好相关性。进一步实验表明，当结合标签层级时，这些指标可有效预测层次分类任务的下游性能。
原文摘要 · Abstract (English)
We introduce two reference-free metrics for quality evaluation of taxonomies in the absence of labels. The first metric evaluates robustness by calculating the correlation between semantic and taxonomic similarity, addressing error types not considered by existing metrics. The second uses Natural Language Inference to assess logical adequacy. Both metrics are tested on five taxonomies and are shown to correlate well with F1 against ground truth taxonomies. We further demonstrate that our metrics can predict downstream performance in hierarchical classification when used with label hierarchies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。