用AI识别医学论文中的夸大宣传语言,提升研究可信度。
Hype or not? Formalizing Automatic Promotional Language Detection in Biomedical Research
- 定义夸大语言并制定可复现的标注规范。
- 在NIH资助申请书上训练模型,准确率接近人类水平。
- 适合关注科研诚信与NLP评估的研究者。
科学领域中夸大性语言('hype')日益增多,可能削弱证据的客观评估,阻碍研究进展,并损害科学公信力。本文提出自动检测夸大语言的任务,将其定义为作者用于美化、推广或夸张研究亮点的夸张或主观表达。我们制定了形式化标注指南,并应用于美国国立卫生研究院(NIH)资助申请书语料库的部分标注。随后评估了传统文本分类器与语言模型在此任务上的表现,与人工基准进行比较。实验表明,形式化标注指南能帮助人类可靠标注候选夸大形容词,利用标注数据训练机器学习模型也取得良好效果。研究揭示了该任务的语言复杂性,以及对领域知识和时间敏感性的潜在需求。尽管已有部分语言学工作涉及夸大检测,但据我们所知,这是首个将此任务作为自然语言处理问题系统研究的工作。
原文摘要 · Abstract (English)
In science, promotional language ('hype') is increasing and can undermine objective evaluation of evidence, impede research development, and erode trust in science. In this paper, we introduce the task of automatic detection of hype, which we define as hyperbolic or subjective language that authors use to glamorize, promote, embellish, or exaggerate aspects of their research. We propose formalized guidelines for identifying hype language and apply them to annotate a portion of the National Institutes of Health (NIH) grant application corpus. We then evaluate traditional text classifiers and language models on this task, comparing their performance with a human baseline. Our experiments show that formalizing annotation guidelines can help humans reliably annotate candidate hype adjectives and that using our annotated dataset to train machine learning models yields promising results. Our findings highlight the linguistic complexity of the task, and the potential need for domain knowledge and temporal awareness of the facts. While some linguistic works address hype detection, to the best of our knowledge, we are the first to approach it as a natural language processing task.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。