推理型大模型能提升临床病历分类准确率，但稳定性较差。
Can Reasoning LLMs Enhance Clinical Document Classification?
- 对比8个大模型，推理类在准确率上优于非推理类。
- 推理模型最高达75%准确率，但一致性仅84%。
- 适合需高精度的复杂病例分类，不适用于追求稳定的场景。
临床文档分类对将非结构化医疗文本转化为标准化ICD-10诊断至关重要，但受限于复杂的医学语言、隐私约束和标注数据稀缺。本研究评估了八种大语言模型（四种推理型：Qwen QWQ、Deepseek Reasoner、GPT o3 Mini、Gemini 2.0 Flash Thinking；四种非推理型：Llama 3.3、GPT 4o Mini、Gemini 2.0 Flash、Deepseek Chat）在MIMIC-IV数据集上的表现。使用cTAKES结构化临床叙述，通过三次实验运行并采用多数投票确定最终预测。结果显示，推理模型在准确率（71% vs 68%）和F1分数（67% vs 60%）上优于非推理模型，其中Gemini 2.0 Flash Thinking达到最高准确率75%和F1分数76%。然而，非推理模型表现出更高的一致性（91% vs 84%）。性能在不同ICD-10编码间差异显著，推理模型在复杂病例中表现更优，但在抽象类别上表现不佳。结果表明存在准确率与一致性的权衡，建议采用混合方法优化临床编码。未来研究应探索多标签分类、领域特定微调和集成方法以提升真实场景下的可靠性。
原文摘要 · Abstract (English)
Clinical document classification is essential for converting unstructured medical texts into standardised ICD-10 diagnoses, yet it faces challenges due to complex medical language, privacy constraints, and limited annotated datasets. Large Language Models (LLMs) offer promising improvements in accuracy and efficiency for this task. This study evaluates the performance and consistency of eight LLMs; four reasoning (Qwen QWQ, Deepseek Reasoner, GPT o3 Mini, Gemini 2.0 Flash Thinking) and four non-reasoning (Llama 3.3, GPT 4o Mini, Gemini 2.0 Flash, Deepseek Chat); in classifying clinical discharge summaries using the MIMIC-IV dataset. Using cTAKES to structure clinical narratives, models were assessed across three experimental runs, with majority voting determining final predictions. Results showed that reasoning models outperformed non-reasoning models in accuracy (71% vs 68%) and F1 score (67% vs 60%), with Gemini 2.0 Flash Thinking achieving the highest accuracy (75%) and F1 score (76%). However, non-reasoning models demonstrated greater stability (91% vs 84% consistency). Performance varied across ICD-10 codes, with reasoning models excelling in complex cases but struggling with abstract categories. Findings indicate a trade-off between accuracy and consistency, suggesting that a hybrid approach could optimise clinical coding. Future research should explore multi-label classification, domain-specific fine-tuning, and ensemble methods to enhance model reliability in real-world applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。