用大模型生成合成数据,自动识别论文中的数据集引用。
Large Language Models and Synthetic Data for Monitoring Dataset Mentions in Research Papers
- 用大模型和合成数据构建弱监督训练集,提升低资源场景下的泛化能力。
- 在人工标注数据上微调后,准确率优于NuExtract-v1.5和GLiNER-large-v2.1。
- 适合研究者、资助方与政策制定者追踪数据使用,提升数据可发现性。
追踪学术论文中对数据集的提及与使用,有助于提升数据可发现性、质量和生产效率。然而,人工识别海量文献中的数据集引用成本高且不可扩展。本文提出一种基于大语言模型(LLMs)、合成数据和两阶段微调的自动化框架,实现跨领域的数据集提及检测。通过零样本提取、大模型作为评估者及推理代理生成弱监督合成数据集,再以Phi-3.5-mini instruct模型进行预微调,并在人工标注子集上进一步微调。推理时采用ModernBERT分类器高效过滤数据集提及,降低计算开销同时保持高召回率。在独立人工标注样本上的评估显示,该模型在数据集抽取准确率上优于NuExtract-v1.5和GLiNER-large-v2.1。结果表明,由大模型生成的合成数据能有效缓解训练数据稀缺问题,在低资源场景下显著提升模型性能。该框架为大规模监测数据使用提供了可行路径,增强透明度,助力研究人员、资助方与政策制定者识别数据缺口,推动数据可及性提升。
原文摘要 · Abstract (English)
Tracking how data is mentioned and used in research papers provides critical insights for improving data discoverability, quality, and production. However, manually identifying and classifying dataset mentions across vast academic literature is resource-intensive and not scalable. This paper presents a machine learning framework that automates dataset mention detection across research domains by leveraging large language models (LLMs), synthetic data, and a two-stage fine-tuning process. We employ zero-shot extraction from research papers, an LLM-as-a-Judge for quality assessment, and a reasoning agent for refinement to generate a weakly supervised synthetic dataset. The Phi-3.5-mini instruct model is pre-fine-tuned on this dataset, followed by fine-tuning on a manually annotated subset. At inference, a ModernBERT-based classifier efficiently filters dataset mentions, reducing computational overhead while maintaining high recall. Evaluated on a held-out manually annotated sample, our fine-tuned model outperforms NuExtract-v1.5 and GLiNER-large-v2.1 in dataset extraction accuracy. Our results highlight how LLM-generated synthetic data can effectively address training data scarcity, improving generalization in low-resource settings. This framework offers a pathway toward scalable monitoring of dataset usage, enhancing transparency, and supporting researchers, funders, and policymakers in identifying data gaps and strengthening data accessibility for informed decision-making.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。