用简单RAG框架解决真实世界事实核查，效果优于传统方法。
AIC CTU system at AVeriTeC: Re-framing automated fact-checking as a simple RAG task
- 将事实核查重构为检索增强生成任务，利用LLM直接输出答案。
- 在AVeriTeC数据集上达到第三名，GPT-4o表现最优。
- 开源代码并支持开放源模型，适合研究与应用落地。
本文介绍了我们在AVeriTeC共享任务中获得第三名的方案，提出一种专为真实世界证据检索设计的简化版检索增强生成（RAG）框架，借助大语言模型的推理能力实现自动事实核查。我们公开了代码库，并详细说明了两个核心模块：检索器与证据及标签生成器，解释了其采用的MMR重排序和李克特量表置信度估计等特性。在AVeriTeC开发集与测试集上评估结果表明，GPT-4o是当时最合适的模型，而Llama 3.1 70B则展现出优秀的开源替代潜力。通过实证误差分析发现，预测错误多与数据噪声或事实本身模糊有关，提示未来需加强数据清洗与增强。
原文摘要 · Abstract (English)
This paper describes our $3^{rd}$ place submission in the AVeriTeC shared task in which we attempted to address the challenge of fact-checking with evidence retrieved in the wild using a simple scheme of Retrieval-Augmented Generation (RAG) designed for the task, leveraging the predictive power of Large Language Models. We release our codebase and explain its two modules - the Retriever and the Evidence & Label generator - in detail, justifying their features such as MMR-reranking and Likert-scale confidence estimation. We evaluate our solution on AVeriTeC dev and test set and interpret the results, picking the GPT-4o as the most appropriate model for our pipeline at the time of our publication, with Llama 3.1 70B being a promising open-source alternative. We perform an empirical error analysis to see that faults in our predictions often coincide with noise in the data or ambiguous fact-checks, provoking further research and data augmentation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。