用合成数据提升大模型识别科学伪论的能力
MisSynth: Improving MISSCI Logical Fallacies Classification with Synthetic Data
- 通过检索增强生成构建伪论合成数据
- 微调后模型F1分数提升超35%
- 适合研究虚假信息检测的学者与开发者
健康类虚假信息广泛存在且危害大，尤其当论点扭曲或误读科学发现时更难识别。本文研究合成数据生成与轻量微调对大语言模型（LLMs）识别谬误能力的影响，基于MISSCI数据集与框架提出MisSynth流水线：利用检索增强生成（RAG）生成合成谬误样本，并用于微调LLM。结果表明，相比原始基线，微调后的模型性能显著提升，例如LLaMA 3.1 8B模型在MISSCI测试集上F1得分绝对提升超过35%。实验显示，引入合成谬误数据可有效增强有限标注资源下的零样本分类性能，即使计算资源有限也能实现显著改进。代码与合成数据集已开源。
原文摘要 · Abstract (English)
Health-related misinformation is very prevalent and potentially harmful. It is difficult to identify, especially when claims distort or misinterpret scientific findings. We investigate the impact of synthetic data generation and lightweight fine-tuning techniques on the ability of large language models (LLMs) to recognize fallacious arguments using the MISSCI dataset and framework. In this work, we propose MisSynth, a pipeline that applies retrieval-augmented generation (RAG) to produce synthetic fallacy samples, which are then used to fine-tune an LLM model. Our results show substantial accuracy gains with fine-tuned models compared to vanilla baselines. For instance, the LLaMA 3.1 8B fine-tuned model achieved an over 35% F1-score absolute improvement on the MISSCI test split over its vanilla baseline. We demonstrate that introducing synthetic fallacy data to augment limited annotated resources can significantly enhance zero-shot LLM classification performance on real-world scientific misinformation tasks, even with limited computational resources. The code and synthetic dataset are available on https://github.com/mxpoliakov/MisSynth.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。