用大模型+搜索实现可解释的谣言验证，效果优于一次性提问。
Multi-hop Evidence Pursuit Meets the Web: Team Papelo at FEVER 2024
- 分步生成问题，逐轮搜索证据，逐步逼近真相。
- 在FEVER 2024上达成0.510的验证分数，比基线高0.155。
- 适合关注可解释性、自动化事实核查的研究者与开发者。
在互联网上区分虚假信息与事实一直考验着人类的搜索与推理能力。我们证明，大型语言模型（LLMs）的推理能力与现代搜索引擎的检索能力相结合，可自动且可解释地验证声明。我们采用多跳证据追寻策略，利用序列到序列模型根据输入声明生成初始问题，通过搜索获取答案并迭代生成后续问题，持续追寻缺失证据。该方法在FEVER 2024（AVeriTeC）共享任务中表现优异：相比一次性生成所有问题的策略，本方法在标签准确率上提升0.045，在AVeriTeC得分上提升0.155（评估证据充分性）。消融实验表明，问题生成方式、中等上下文大小、一次只处理一篇文档、添加元数据、重述问题、将问题简化为二分类以及重新评估最终结论等设计至关重要。提交系统在开发集上取得0.510的AVeriTeC得分，在测试集上取得0.477的得分。
原文摘要 · Abstract (English)
Separating disinformation from fact on the web has long challenged both the search and the reasoning powers of humans. We show that the reasoning power of large language models (LLMs) and the retrieval power of modern search engines can be combined to automate this process and explainably verify claims. We integrate LLMs and search under a multi-hop evidence pursuit strategy. This strategy generates an initial question based on an input claim using a sequence to sequence model, searches and formulates an answer to the question, and iteratively generates follow-up questions to pursue the evidence that is missing using an LLM. We demonstrate our system on the FEVER 2024 (AVeriTeC) shared task. Compared to a strategy of generating all the questions at once, our method obtains .045 higher label accuracy and .155 higher AVeriTeC score (evaluating the adequacy of the evidence). Through ablations, we show the importance of various design choices, such as the question generation method, medium-sized context, reasoning with one document at a time, adding metadata, paraphrasing, reducing the problem to two classes, and reconsidering the final verdict. Our submitted system achieves .510 AVeriTeC score on the dev set and .477 AVeriTeC score on the test set.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。