自动提取和生成学术论文中的局限性，提升研究透明度。
BAGELS: Benchmarking the Automated Generation and Extraction of Limitations from Scholarly Text
- 用检索增强生成技术自动生成论文局限性
- 构建了来自三大顶会的局限性数据集
- 提供细粒度评估框架，适合论文审核与科研诚信研究
在科学研究中，'局限性'指研究的不足、约束或弱点。透明报告这些局限性有助于提升研究质量、可复现性，并增强公众对科学的信任。然而，作者常低估局限性报告，依赖模糊表述应付编辑要求，损害读者理解与信心。这一现象叠加科研论文数量激增，亟需自动化方法从学术文本中提取并生成局限性。为此，本文提出一套完整的计算分析架构：(1) 从ACL、NeurIPS和PeerJ论文中提取局限性，并结合外部评审补充数据，构建局限性数据集；(2) 提出基于新型检索增强生成（RAG）的技术，自动生成局限性；(3) 设计细粒度评估框架，并对生成方法进行元评估。该工作为提升科研透明度提供了可扩展的技术路径。
原文摘要 · Abstract (English)
In scientific research, ``limitations'' refer to the shortcomings, constraints, or weaknesses of a study. A transparent reporting of such limitations can enhance the quality and reproducibility of research and improve public trust in science. However, authors often underreport limitations in their papers and rely on hedging strategies to meet editorial requirements at the expense of readers' clarity and confidence. This tendency, combined with the surge in scientific publications, has created a pressing need for automated approaches to extract and generate limitations from scholarly papers. To address this need, we present a full architecture for computational analysis of research limitations. Specifically, we (1) create a dataset of limitations from ACL, NeurIPS, and PeerJ papers by extracting them from the text and supplementing them with external reviews; (2) we propose methods to automatically generate limitations using a novel Retrieval Augmented Generation (RAG) technique; (3) we design a fine-grained evaluation framework for generated limitations, along with a meta-evaluation of these techniques.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。