在本地硬件上评估大模型生成医药制造SQL的能力，发现通用代码模型优于专用医学模型。
Benchmarking Local LLMs for Natural-Language-to-SQL Querying in Biopharmaceutical Manufacturing: An Empirical Benchmark on Consumer-Grade Hardware
- 用本地部署的四个开源大模型生成医药制造数据库的SQL语句
- 主流模型在60个专业问题上均能正确生成SQL，Qwen和Llama表现最佳
- 适合需数据隐私保护的医药企业，但需人工审核以满足监管要求
生物制药制造机构受FDA指南、欧盟GMP及欧盟人工智能法案等监管框架约束，难以使用云端AI系统。本地部署的大语言模型（LLMs）提供了隐私保护替代方案，但其在制药制造任务中的适用性尚未充分研究。本研究评估了四种开源LLM（Qwen 2.5 Coder 7B、Llama 3.1 8B、Mistral 7B、Meditron 7B）通过Ollama在本地部署后，对制药制造数据库进行自然语言转SQL生成的能力。基于FastAPI构建的PharmaBatchDB AI平台，使用包含约6.3万条记录的合成Microsoft SQL Server数据库，涵盖批次、制造执行系统（MES）和在线清洗（CIP）模块。采用包括SQL提取率、合规性、事实一致性、ROUGE-L、幻觉率、吞吐量与延迟在内的多项指标，在60个领域特定自然语言问题上进行评估。Qwen 2.5 Coder 7B、Llama 3.1 8B和Mistral 7B成功完成所有任务，而Meditron 7B因上下文窗口限制和生成能力差，几乎无法完成任务。Llama 3.1 8B达到最高SQL合规性，Qwen 2.5 Coder 7B在文本相似性和事实一致性方面最优。两者性能差异无统计显著性。结果表明，经过代码微调的通用大模型在结构化查询生成上优于领域专用生物医学模型。尽管完全本地化且符合GxP标准，当前系统仍需人工监督与下游验证才能用于受监管场景。
原文摘要 · Abstract (English)
Biopharmaceutical manufacturing organizations operate under regulatory frameworks such as FDA guidance, EU Good Manufacturing Practice (GMP), and the EU AI Act, which can restrict the use of cloud-based artificial intelligence systems. Locally deployed large language models (LLMs) offer a privacy-preserving alternative, but their suitability for pharmaceutical manufacturing tasks remains underexplored. This study evaluates four open-source LLMs (Qwen 2.5 Coder 7B, Llama 3.1 8B, Mistral 7B, and Meditron 7B) deployed locally via Ollama for natural-language-to-SQL generation over a pharmaceutical manufacturing database. A FastAPI-based evaluation platform, PharmaBatchDB AI, was developed using a synthetic Microsoft SQL Server database containing approximately 63,000 records across Batch, Manufacturing Execution System (MES), and Clean-In-Place (CIP) modules. Models were benchmarked on 60 domain-specific natural-language questions using metrics including SQL extraction rate, SQL compliance, factual consistency, ROUGE-L, hallucination rate, throughput, and latency. Qwen 2.5 Coder 7B, Llama 3.1 8B, and Mistral 7B generated SQL for all evaluation tasks, while Meditron 7B failed on nearly all tasks due to context-window limitations and poor SQL generation capability. Llama 3.1 8B achieved the highest SQL compliance, whereas Qwen 2.5 Coder 7B achieved the strongest overall text similarity and factual consistency. Performance differences between the two leading models were not statistically significant. The results show that code-tuned general-purpose LLMs outperform a domain-specific biomedical model on structured query generation for pharmaceutical manufacturing data. Although fully local, GxP-aligned NLQ systems are feasible on consumer hardware, current performance levels still require human oversight and downstream validation for regulated use.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。