用轻量校准让小模型达到大模型的查证准确率，降低成本。
Calibrating Small Language Models for Claim Check-Worthiness Detection
- 推理时加校准层，不重训练就能提升小模型性能。
- 小模型F1提升12%至33.80%，媲美大模型。
- 适合资源有限但需高准确率的部署场景。
判断声明是否值得核查是自动化事实核查流程的关键第一步。本文源于一家早期初创公司的实际部署挑战：对每条新声明运行大语言模型（LLM）成本和延迟过高，而小模型（SLM）则牺牲了准确性。我们提出NN-PPI，即预测驱动推断（PPI）的点式扩展，通过在推理时添加轻量级后处理校准层，无需重训练底层模型即可提升预测质量。实验表明，根据基线模型规模与性能不同，NN-PPI带来12%至33.80%的加权F1提升，使小模型性能逼近大模型水平。除少样本小模型外，该方法还显著提升了已上线的微调模型表现，证明残差校准与监督微调具有互补性。通过从仅需其十分之一服务成本的模型中恢复出接近LLM级别的准确性，使得大规模核查检测的运营成本大幅降低。代码与数据见https://anonymous.4open.science/r/arr-claim-worthiness-F237。
原文摘要 · Abstract (English)
Assessing claim check-worthiness is an essential first step in automated fact-checking pipelines. This work is motivated by a real deployment challenge at an early-stage startup: running large language models (LLMs) over every incoming claim is cost- and latency-prohibitive, yet smaller models sacrifice accuracy. We propose NN-PPI, a pointwise extension of Prediction-Powered Inference (PPI) that calibrates model predictions at inference time as a lightweight post-hoc layer, without re-training the underlying model. NN-PPI achieves weighted F1 gains ranging from 12% to 33.80% depending on the size and performance of the baseline model, bringing SLMs on par with larger LLMs. Beyond few-shot SLMs, NN-PPI further improves a production-deployed fine-tuned model, demonstrating that residual calibration is complementary to supervised fine-tuning. By recovering LLM-level accuracy from models that are an order of magnitude cheaper to serve, it makes accurate check-worthiness detection substantially cheaper to operate at scale. Our code and data can be found at https://anonymous.4open.science/r/arr-claim-worthiness-F237.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。