用开源小模型评估医疗共决，兼顾隐私与可持续性。
Measuring the practice of shared-decision making (OPTION12): An Investigation into Open-sourced Smaller LLMs (OS-sLLMs) for Better Privacy and Sustainability
- 采用开源小模型+专家标注数据，本地部署评估共决行为。
- Gemma3:12b表现最佳，与人工标注相关性达0.59（斯皮尔曼）。
- 适合关注隐私保护、需人机协同的医疗决策研究者。
我们提出LLM4SDM，首个基于开源小型语言模型（OS-sLLMs）的自动化共享决策（SDM）评估研究，采用观察者OPTION12框架。不同于依赖大型商业模型和简短OPTION5工具的前人工作，本研究聚焦于隐私保护、可本地部署的模型，使用荷兰黑色素瘤咨询对话数据进行验证。基于专家标注的临床会话，我们在开发阶段试点评估了三个通用领域和两个医学领域OS-sLLMs。结果显示，通用领域模型表现优于医学领域模型，后者存在严重幻觉和指令遵循失败问题。其中，Gemma3:12b与人工标注的最高一致性为皮尔逊相关系数0.51，斯皮尔曼等级相关系数0.59。项目级与定性分析揭示了时间话语推理、对话角色归属及证据定位等系统性挑战。为此，我们引入判官-大模型共识框架以解决多模型分歧。结果表明，当前OS-sLLMs尚无法替代人类标注员，但为隐私友好型人机协同的SDM评估提供了有前景的基础。
原文摘要 · Abstract (English)
We present LLM4SDM, the first study of open-source smaller language models (OS-sLLMs) for automated assessment of shared decision making (SDM) using the Observer OPTION12 framework. Unlike previous work that relies on large commercial models and the shorter OPTION5 instrument, our study focuses on privacy-preserving locally deployable models and Dutch melanoma consultation transcripts. Using expert-annotated clinical consultations, we evaluate three general-domain and two medical-domain OS-sLLMs during a development-phase pilot study. Results show that general-domain models outperform medical-domain models, which exhibit substantial hallucination and instruction-following failures. Gemma3:12b achieves the strongest agreement with human annotations (Pearson r=0.51, Spearman \r{ho}=0.59). Item-level and qualitative analyses reveal systematic challenges related to temporal discourse reasoning, conversational role attribution, and evidence grounding. We further introduce a Judge-LLM consensus framework designed to support disagreement resolution among multiple models. Our findings suggest that while current OS-sLLMs cannot replace human annotators, they offer a promising foundation for privacy-preserving human-in-the-loop SDM assessment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。