提出自适应标签的自我解释方法,提升事实验证准确率与解释质量。
Take It Easy: Label-Adaptive Self-Rationalization for Fact Verification and Explanation Generation
- 分步微调:先学判断真假,再学生成关联证据的解释。
- 在两个数据集上准确率提升超10个百分点,优于GPT-4。
- 用合成解释少量训练即可达到接近全量标注的效果,成本低。
为帮助记者进行事实核查,现有计算方法常需适配特定领域并生成解释。然而多数自动核查方法依赖三分类数据集,难以反映真实误信息场景;且解释多基于证据摘要,未能体现论断与证据间的关系。为此,我们将通常用于自然语言推理(NLI)的自我解释方法扩展至事实验证任务。提出标签自适应学习策略:首先在带标签数据上微调模型以预测真伪(步骤1模型),随后在同一数据集及附加解释标注下再次微调,实现自我解释。结果表明,该方法在PubHealth和AVeriTec数据集上均使宏观F1提升超过10个百分点,优于GPT-4。此外,为降低解释标注成本,我们从GPT-4-turbo、GPT-3.5-turbo和Llama-3-8B生成64条合成解释,并采用少样本微调步骤1模型。该模型性能接近全量标注微调版本,证明了合成数据在低成本可解释事实核查中的潜力。该方法为不同标注体系下的真实世界可解释事实核查提供了新方向。
原文摘要 · Abstract (English)
Computational methods to aid journalists in the task often require adapting a model to specific domains and generating explanations. However, most automated fact-checking methods rely on three-class datasets, which do not accurately reflect real-world misinformation. Moreover, fact-checking explanations are often generated based on text summarization of evidence, failing to address the relationship between the claim and the evidence. To address these issues, we extend the self-rationalization method--typically used in natural language inference (NLI) tasks--to fact verification. We propose a label-adaptive learning approach: first, we fine-tune a model to learn veracity prediction with annotated labels (step-1 model). Then, we fine-tune the step-1 model again to learn self-rationalization, using the same data and additional annotated explanations. Our results show that our label-adaptive approach improves veracity prediction by more than ten percentage points (Macro F1) on both the PubHealth and AVeriTec datasets, outperforming the GPT-4 model. Furthermore, to address the high cost of explanation annotation, we generated 64 synthetic explanations from three large language models: GPT-4-turbo, GPT-3.5-turbo, and Llama-3-8B and few-shot fine-tune our step-1 model. The few-shot synthetic explanation fine-tuned model performed comparably to the fully fine-tuned self-rationalization model, demonstrating the potential of low-budget learning with synthetic data. Our label-adaptive self-rationalization approach presents a promising direction for future research on real-world explainable fact-checking with different labeling schemes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。