评估医疗语言模型性能，避免盲目依赖测试集指标。
Performance Assessment Strategies for Language Model Applications in Healthcare
- 结合临床任务特点设计评估方法。
- 指出传统基准测试易过拟合，泛化能力差。
- 推荐用专家评估和低成本模型辅助验证。
语言模型（LMs）在人工智能领域正成为新兴范式，广泛应用于医疗体系。要有效评估其应用表现，需深入理解临床任务特性，并关注实际临床环境中性能的可变性。当前主流评估方法依赖量化基准，但存在局限性，可能因训练对测试集过拟合，导致模型在其他任务或数据分布上泛化能力下降。因此，融合人类专家判断与低成本计算模型作为评估器的策略日益受到关注。本文综述了当前医疗领域语言模型应用评估的先进方法，涵盖医疗设备中的应用场景。
原文摘要 · Abstract (English)
Language models (LMs) represent an emerging paradigm within artificial intelligence, with applications throughout the medical enterprise. A comprehensive understanding of the clinical task and awareness of the variability in performance when implemented in actual clinical environments lays the foundation for the LM application assessment. Presently, a prevalent method for evaluating the performance of these generative models relies on quantitative benchmarks. Such benchmarks have limitations and may suffer from train-to-the-test overfitting, optimizing performance for a specified test set at the cost of generalizability across other tasks and data distributions. Evaluation strategies leveraging human expertise and utilizing cost-effective computational models as evaluators are gaining interest. We discuss current state-of-the-art methodologies for assessing the performance of LM applications in healthcare and medical devices.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。