构建科学文献定义提取的基准与工具链，助力术语标准化。
SciDef: Datasets and Tools for Automated Definition Extraction from Scientific Literature with LLMs
- 提出DefExtra等三类数据集与自动化处理流水线。
- 最强配置准确率达0.397，覆盖率达86.4%但存在过生成。
- 适用于需术语统一的科研分析与知识图谱构建者。
科学概念在不同论文中常表述不一，影响研究比较、术语复用及下游资源构建。我们提出SciDef，一套面向科学定义提取的资源套件，包含DefExtra（268条人工验证的作者陈述定义，来自75篇论文）、DefSim（60对人工标注的定义相似度判断）以及一个开源的基于LLM的PDF预处理、分块、定义提取、提示优化与评估流水线。通过在16个语言模型上测试不同提示策略与分块方案，最强的集级配置得分0.397；最高覆盖率配置能匹配至少一条预测至86.4%的黄金定义，但存在过生成问题。进一步表明，基于NLI的匹配指标与人工判别高度一致。这些结果使SciDef成为可复用的基准与工具层，同时揭示了相关性感知过滤是全自动定义提取的关键瓶颈。代码与数据集见https://github.com/Media-Bias-Group/SciDef。
原文摘要 · Abstract (English)
Scientific concepts are often defined inconsistently across papers, making it difficult to compare findings, reuse terminology, and build reliable downstream resources. We present SciDef, a resource suite for scientific definition extraction. The suite contains DefExtra, a benchmark of 268 human-validated author-stated definitions from 75 academic papers; DefSim, 60 human-labeled definition-pair similarity judgments; and an open LLM-based pipeline for PDF preprocessing, chunking, definition extraction, prompt optimization, and evaluation. We validate the resources by benchmarking 16 language models across prompting strategies and chunking schemes. The strongest set-level configuration achieves a score of 0.397, while the highest-coverage configuration matches at least one prediction to 86.4% of gold definitions but over-generates candidate definitions. We further show that an NLI-based matching metric agrees strongly with human DefSim judgments. These results position SciDef as a reusable benchmark and tooling layer for definition-centric literature analysis, while highlighting relevance-aware filtering as the key bottleneck for fully automatic definition extraction. Code & datasets are available at https://github.com/Media-Bias-Group/SciDef.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。