用合成数据微调模型，6.6倍提速长文本事实性评估
VeriFastScore: Speeding up long-form factuality evaluation
- 用合成数据微调Llama3.1 8B，一次完成文本中所有可验证陈述的提取与验证
- 与原VeriScore相比，例子级相关性0.80，系统级0.94，速度提升6.6倍
- 适合需要大规模事实性评估的研究者和实际应用开发人员
FactScore和VeriScore等指标通过将输入文本分解为原子陈述并逐一验证来评估长文本的事实性。尽管有效且可解释，但这类方法需大量LLM调用，单次评估耗时长达100秒，限制了其在大规模评估和训练中的实用性。为此，我们提出VeriFastScore，利用合成数据微调Llama3.1 8B模型，基于Google Search获取的证据，同时完成文本中所有可验证陈述的提取与验证。我们发现，仅靠提示工程无法解决该任务：模型平均接收约4K tokens证据，需同步完成陈述分解、可验证性判断与验证。然而，微调后的VeriFastScore模型在例子级（r=0.80）和系统级（r=0.94）与原管道高度相关，整体提速6.6倍（排除证据检索后达9.9倍）。为促进未来研究，我们公开发布VeriFastScore模型及合成数据集。
原文摘要 · Abstract (English)
Metrics like FactScore and VeriScore that evaluate long-form factuality operate by decomposing an input response into atomic claims and then individually verifying each claim. While effective and interpretable, these methods incur numerous LLM calls and can take upwards of 100 seconds to evaluate a single response, limiting their practicality in large-scale evaluation and training scenarios. To address this, we propose VeriFastScore, which leverages synthetic data to fine-tune Llama3.1 8B for simultaneously extracting and verifying all verifiable claims within a given text based on evidence from Google Search. We show that this task cannot be solved via few-shot prompting with closed LLMs due to its complexity: the model receives ~4K tokens of evidence on average and needs to concurrently decompose claims, judge their verifiability, and verify them against noisy evidence. However, our fine-tuned VeriFastScore model demonstrates strong correlation with the original VeriScore pipeline at both the example level (r=0.80) and system level (r=0.94) while achieving an overall speedup of 6.6x (9.9x excluding evidence retrieval) over VeriScore. To facilitate future factuality research, we publicly release our VeriFastScore model and synthetic datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。