从同行评审中构建科学问答数据集，支持真实场景的文本理解与回答。
PeerQA: A Scientific Question Answering Dataset from Peer Reviews
- 从真实论文评审中提取问题，由作者亲自标注答案。
- 覆盖579组问答，平均文章长度12000词，适合长文本建模测试。
- 揭示去上下文化能显著提升文档级检索效果，适合研究者使用。
我们提出 PeerQA，一个源自同行评审的真实世界科学文档级问答数据集。该数据集的问题来自审稿人对学术论文的深入审查过程，答案由原作者标注。共包含579个问答对，来自208篇学术论文，主要集中在机器学习和自然语言处理领域，也涵盖地球科学、公共卫生等其他学科。PeerQA 支持三项关键任务：证据检索、不可回答问题分类和答案生成。我们提供了详细的数据分析，并为三项任务建立了基线系统。实验表明，在文档级检索中，即使采用简单的去上下文化方法，也能在不同架构上持续提升性能。在答案生成任务中，由于论文平均长度达12000个词，PeerQA 成为长上下文建模的挑战性基准。代码与数据已公开于 https://github.com/UKPLab/peerqa。
原文摘要 · Abstract (English)
We present PeerQA, a real-world, scientific, document-level Question Answering (QA) dataset. PeerQA questions have been sourced from peer reviews, which contain questions that reviewers raised while thoroughly examining the scientific article. Answers have been annotated by the original authors of each paper. The dataset contains 579 QA pairs from 208 academic articles, with a majority from ML and NLP, as well as a subset of other scientific communities like Geoscience and Public Health. PeerQA supports three critical tasks for developing practical QA systems: Evidence retrieval, unanswerable question classification, and answer generation. We provide a detailed analysis of the collected dataset and conduct experiments establishing baseline systems for all three tasks. Our experiments and analyses reveal the need for decontextualization in document-level retrieval, where we find that even simple decontextualization approaches consistently improve retrieval performance across architectures. On answer generation, PeerQA serves as a challenging benchmark for long-context modeling, as the papers have an average size of 12k tokens. Our code and data is available at https://github.com/UKPLab/peerqa.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。