用大模型排序推理路径,提升多语言数值声明验证准确率
DS@GT ARC at CheckThat! 2026: LLM-Based Trace Ranking and Grouped Reward Modeling for Multilingual Numerical Claim Verification

- 用LoRA微调大模型独立评分推理链,选最优结果
- 阿拉伯语专用模型AraBERT优于通用多语言模型
- 轻量级奖励模型在矛盾类上表现更好,适合特定场景
自动化数值声明验证需兼顾语言理解与定量推理。本文介绍我们在CLEF 2026 CheckThat!任务2中的系统,针对英语和阿拉伯语的数值声明进行推理路径排序与最终结论预测。提出两种方法:第一种基于LoRA微调大模型,将每条推理路径作为二分类问题评分,并采用Best-of-N选择最终结论;进一步尝试自适应子声明分解以简化复杂声明。第二种方法采用轻量级TF-IDF奖励模型,结合手工设计的数值与时间重叠特征评分,按结论分组聚合得分。在阿拉伯语上对比通用多语言模型与专为阿拉伯语预训练的AraBERT模型。结果表明,大模型方法在多数指标上更优,尤其在Recall@5上表现突出;奖励模型在矛盾类上更具优势。子声明分解未提升性能,可能引入噪声。AraBERT在多数指标上优于通用基线。
原文摘要 · Abstract (English)
Automated verification of numerical claims is a challenging problem, as it requires both language understanding and quantitative reasoning. This paper describes our system for CLEF 2026 CheckThat! Task 2, which focuses on ranking reasoning traces generated by large language models (LLMs) and predicting a final verdict for numerical claims in English and Arabic. We explore two approaches. The first approach fine-tunes an LLM-based verifier using LoRA to score each reasoning trace independently as a binary classification problem, and selects the final verdict using Best-of-N selection. We further experiment with adaptive sub-claim decomposition to break complex claims into simpler parts before verification. The second approach uses a lightweight TF-IDF reward model with handcrafted numeric and temporal overlap features to score traces, and aggregates scores by verdict group to determine the final prediction. For Arabic, we compare a general multilingual model against AraBERT, a language-specific model pretrained on Arabic text. Our results show that the LLM-based approach outperforms the lightweight reward model on most metrics, particularly Recall@5, while the reward-based approach shows stronger performance on the Conflicting class. Sub-claim decomposition did not improve performance, suggesting that claim splitting introduces noise rather than aiding reasoning. For Arabic, AraBERT outperforms the multilingual baseline across most metrics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。