用原文片段精准回答学术问题，避免大模型胡说八道
ACL-Verbatim: hallucination-free question answering for research
- 直接从论文中提取原文片段作为答案，杜绝幻觉
- 150万参数模型在关键词级F1达53.6，优于最强LLM提取器
- 专为学术问答设计，适合研究人员快速查文献
学术研究需要高效可靠的高质量信息获取方法，但当前基于大语言模型（LLMs）的辅助工具仍存在生成事实错误或无意义内容的问题，即幻觉。本文将可解释的抽取式问答系统VerbatimRAG应用于ACL Anthology中的研究论文，将用户查询直接映射到检索文档中的原文片段。我们构建了一个新的基准数据集，用于将用户查询映射到研究论文的相关文本段落，并基于此训练和评估多种抽取模型。人工标注由自然语言处理研究人员完成，使用基于ScIRGen方法的定制化流水线生成合成用户查询，并与VerbatimRAG检索到的论文片段配对。在该基准上，使用我们流水线生成的银标签进行监督训练的150M参数ModernBERT词级分类器达到最佳词级F1（53.6），优于最强评估的LLM抽取器（48.7）。
原文摘要 · Abstract (English)
Academic researchers need efficient and reliable methods for collecting high-quality information from trusted sources, but modern tools for AI-assisted research still suffer from the tendency of Large Language Models (LLMs) to produce factually inaccurate or nonsensical output, commonly referred to as hallucinations. We apply the extractive question answering system VerbatimRAG to research papers in the ACL Anthology, directly mapping user queries to verbatim text spans in retrieved documents. We contribute a novel ground truth dataset for the task of mapping user queries to relevant text spans in research papers, and use it to train and evaluate a variety of extractive models. Human annotation is performed by NLP researchers and is based on synthetic user queries generated using a custom pipeline based on the ScIRGen methodology, paired with chunks of research papers retrieved by VerbatimRAG. On this benchmark, a 150M-parameter ModernBERT token classifier trained on silver supervision from our pipeline achieves the best word-level F1 (53.6), ahead of the strongest evaluated LLM extractor (48.7).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。