小模型经微调后可低成本高效验证生物医学声明真伪。
Small LLMs for Biomedical Claim Verification: Cost-Effective Fine-Tuning, Structural Dataset Shortcuts, and Cross-Domain Generalization
- 用QLoRA微调3个小型模型，仅需1008条数据。
- Mistral-7B在跨领域任务中比GPT-4o高12%准确率。
- 发现数据集结构缺陷，揭示训练数据质量关键作用。
大型语言模型如GPT-4o和GPT-5在生物医学声明验证上表现优异，但成本高且不可解释，限制了大规模应用。本文对三个小型LLM（Phi-3-mini, 3.8B；Qwen2.5-3B；Mistral-7B）采用QLoRA进行微调，基于SciFact与HealthVer数据集，首次对比了QLoRA模型与GPT-4o及微调后的BioLinkBERT编码器。结果显示，仅使用1,008条训练样本，Mistral-7B QLoRA在性能上超越GPT-4o和GPT-5（F1提升最高达12%），且成本极低。通过跨域评估（在SciFact上训练，测试于HealthVer，反之亦然），在模型规模一致下分离出数据量与数据结构的影响。研究发现SciFact中存在未报告的结构偏差，导致域内评分虚高，并证明在结构合理数据上训练可实现稳健的跨域迁移。所有代码与适配器权重将公开发布。
原文摘要 · Abstract (English)
Large Language Models such as GPT-4o and GPT-5 achieve strong zero-shot performance on biomedical claim verification, but cost and opacity limit scalable use. We fine-tune three small LLMs: Phi-3-mini (3.8B), Qwen2.5-3B, and Mistral-7B, via QLoRA on SciFact and HealthVer, providing the first study of QLoRA models against GPT-4o and fine-tuned BioLinkBERT encoders. Mistral-7B QLoRA surpasses both GPT-4o and GPT-5 (up to 12% F1 gain) at a fractional cost using just 1,008 training examples. We conduct extensive in-domain and cross-domain evaluation: models trained on SciFact tested on HealthVer and vice versa, at matched sizes to isolate dataset structure from data quantity. We identify a previously unreported structural artifact in SciFact that inflates in-domain scores, and show through bidirectional out-of-domain evaluation that training on structurally sound data enables robust cross-domain transfer. We plan to release all code and adapter checkpoints.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。