用低秩微调提升医疗对话模型,发现自动评估指标与大模型评判结果不一致。
When Metrics Disagree: Automatic Similarity vs. LLM-as-a-Judge for Clinical Dialogue Evaluation
- 用LoRA技术微调Llama-2-7B,保留基础知识同时适配临床对话
- 自动指标显示改进,但GPT-4评价却更偏好原始模型的对话流畅性
- 提醒:自动化评估不能替代医生专家的实际验证
随着大语言模型(LLMs)在医疗领域应用增多,其可靠性成为关键挑战。现有研究指出通用模型在临床场景中常产生误导性建议。本研究聚焦于使用低秩适配(LoRA)技术对Llama-2-7B进行领域适配,通过向Transformer层注入可训练低秩矩阵,在保留基模型知识的同时,利用真实医患对话数据高效微调。目标是提升模型回答医疗问题的准确性和语境相关性。由于大规模人工评估成本高昂,采用双轨评估框架:轨道A使用传统词汇相似度指标(如BLEU、ROUGE),轨道B则采用GPT-4作为语义评判者。结果显示,经LoRA微调的模型在所有词汇相似度指标上均有显著提升,但在GPT-4评估中却略逊于基线模型的对话流畅性。这一指标分歧揭示:传统自动化评分未必反映实际临床价值。因此,我们主张尽管自动化指标和大模型评判可作为开发辅助,但医疗级部署仍需依赖人类医学专家的严格验证。
原文摘要 · Abstract (English)
As Large Language Models (LLMs) are increasingly integrated into healthcare to address complex inquiries, ensuring their reliability remains a critical challenge. Recent studies have highlighted that generic LLMs often struggle in clinical contexts, occasionally producing misleading guidance. To mitigate these risks, this research focuses on the domain-specific adaptation of \textbf{Llama-2-7B} using the \textbf{Low-Rank Adaptation (LoRA)} technique. By injecting trainable low-rank matrices into the Transformer layers, we efficiently adapted the model using authentic patient-physician transcripts while preserving the foundational knowledge of the base model. Our objective was to enhance precision and contextual relevance in responding to medical queries by capturing the specialized nuances of clinical discourse. Due to the resource-intensive nature of large-scale human validation, the model's performance was evaluated through a dual-track framework: \textbf{Track A} utilized traditional lexical similarity metrics (e.g., BLEU, ROUGE), while \textbf{Track B} employed an "LLM-as-a-Judge" paradigm using GPT-4 for semantic assessment. Our results demonstrate that while the LoRA-enhanced model achieved significant improvements across all quantitative lexical dimensions, a profound disagreement surfaced in the GPT-4 evaluation, which marginally favored the baseline model's conversational flow. This metric divergence underscores a pivotal finding: traditional automated scores may not fully reflect clinical utility. Consequently, we propose that while automated metrics and LLM judges serve as valuable developmental proxies, rigorous validation by human medical experts remains an indispensable requirement for the safe deployment of LLMs in healthcare settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。