测试开源大模型在真实精神科访谈中的表现,发现结构化提示比推理增强更有效。
LLAMADRS: Evaluating Open-Source LLMs on Real Clinical Interviews--To Reason or Not to Reason?
- 用分步评估症状再汇总的ItS策略,误差显著低于直接预测总分
- 25个模型中,强模型在条目级准确率上误差低于临床可接受阈值
- 提示工程设计比模型是否带推理更关键,长推理链效果更好
大型语言模型在众多NLP基准上表现优异,但在真实世界半结构化预测任务中行为仍不明确。我们构建了LlaMADRS,基于精神科访谈语料库CAMI,包含541个会话和5,804条专家标注,用于结构化临床评估。评估了25个开源模型(0.6B至400B参数),生成超过40万条预测。结果表明,高性能开源模型在条目级准确率上残差误差低于临床可接受阈值。此外,分步评估症状后汇总的Item-then-Sum(ItS)策略,在多数模型架构和规模下均显著降低误差,优于直接预测总分(DTS)。尽管推理模型在推理轨迹中尝试分解,但性能提升主要依赖提示设计:标准模型若配备结构化任务定义与示例,可媲美推理增强型。其中,较长推理链对应更低误差;模型规模越大,整体表现越优。研究澄清了推理的有效场景,为半结构化临床评估中部署大模型提供实证指导。
原文摘要 · Abstract (English)
Large language models (LLMs) excel on many NLP benchmarks, but their behavior on real-world, semi-structured prediction remains underexplored. We present LlaMADRS, a benchmark for structured clinical assessment from dialogue built on the CAMI corpus of psychiatric interviews, comprising 5,804 expert annotations across 541 sessions. We evaluate 25 open-source models (standard and reasoning-augmented; 0.6B--400B parameters) and generate over 400,000 predictions. Our results demonstrate that strong open-source LLMs achieve item-level accuracy with residual error below clinically substantial thresholds. Additionally, an Item-then-Sum (ItS) strategy, assessing symptoms individually through discrete LLM calls before synthesizing final scores, significantly reduces error relative to Direct Total Score (DTS) prediction across most model architectures and scales, despite reasoning models attempting similar decomposition in the reasoning traces of their DTS predictions. In fact, we find that performance gains attributed to "reasoning" depend fundamentally on prompt design: standard models equipped with structured task definitions and examples match reasoning-augmented counterparts. Among the latter, longer reasoning traces correlate with reduced error; while higher model scale does across both architectures. Our results clarify when and why reasoning helps and offer actionable guidance for deploying LLMs in semi-structured clinical assessment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。