构建首个大规模斯洛伐克关键词提取数据集,提升低资源语言研究
SlovKE: A Large-Scale Dataset and LLM Evaluation for Slovak Keyphrase Extraction
- 构建22万条斯洛伐克论文摘要数据集,规模达此前25倍
- 基于大模型的KeyLLM方法在精确匹配上表现优于传统方法
- 揭示形态错配是统计方法失效主因,适合多语种信息抽取研究者
针对形态丰富、资源稀缺的斯洛伐克语,构建包含227,432篇科学摘要及作者标注关键词的数据集,数据量较现有最大资源扩大25倍,接近英文基准KP20K规模。利用该数据集,评估了三种无监督基线(YAKE、TextRank、KeyBERT+SlovakBERT)与基于GPT-3.5-turbo的KeyLLM方法。无监督方法在精确匹配上的最高F1@6仅为11.6%,而部分匹配可达51.5%,反映词形变化导致的表面形式不匹配问题。KeyLLM缩小了精确与部分匹配差距,生成更接近作者标准形式的关键词;对100篇文档的手动评估显示其一致性κ=0.61,证实其能捕捉自动化精确匹配遗漏的相关概念。分析表明,形态错配是统计方法的主要失败原因,对其他屈折语言具有参考价值。数据集与代码已公开于Hugging Face与GitHub。
原文摘要 · Abstract (English)
Keyphrase extraction for morphologically rich, low-resource languages remains understudied, largely due to the scarcity of suitable evaluation datasets. We address this gap for Slovak by constructing a dataset of 227,432 scientific abstracts with author-assigned keyphrases -- scraped and systematically cleaned from the Slovak Central Register of Theses -- representing a 25-fold increase over the largest prior Slovak resource and approaching the scale of established English benchmarks such as KP20K. Using this dataset, we benchmark three unsupervised baselines (YAKE, TextRank, KeyBERT with SlovakBERT embeddings) and evaluate KeyLLM, an LLM-based extraction method using GPT-3.5-turbo. Unsupervised baselines achieve at most 11.6\% exact-match $F1@6$, with a large gap to partial matching (up to 51.5\%), reflecting the difficulty of matching inflected surface forms to author-assigned keyphrases. KeyLLM narrows this exact--partial gap, producing keyphrases closer to the canonical forms assigned by authors, while manual evaluation on 100 documents ($κ= 0.61$) confirms that KeyLLM captures relevant concepts that automated exact matching underestimates. Our analysis identifies morphological mismatch as the dominant failure mode for statistical methods -- a finding relevant to other inflected languages. The dataset (https://huggingface.co/datasets/NaiveNeuron/SlovKE) and evaluation code (https://github.com/NaiveNeuron/SlovKE) are publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。