arXiv:2502.08669cs.CL2025-02被引 1

用大模型检测并修复医疗文本错误,发现10%以上错误率会显著降低模型性能。

Assessing the Impact of the Quality of Textual Data on Feature Representation and Machine Learning Models

  • 用Mixtral模型在词级别评估并修正医疗文本错误
  • 错误率超10%时特征表示性能明显下降
  • 适合关注真实世界数据质量的医疗AI研究者

背景:受控环境下收集的数据通常质量较高,但实际应用中常因数据采集问题导致质量下降。已有研究表明数据质量对机器学习模型性能影响显著。方法:提出一种基础错误率指标,在词级别评估文本数据质量;使用Mixtral大型语言模型对低质量数据进行错误检测与纠正。研究对比了高质量的MIMIC-III公共医院数据集和低质量的澳大利亚养老院(ACH)私有数据集。在MIMIC上系统引入不同比例错误,同时利用大模型提升ACH数据质量。结果:从MIMIC和ACH中分别抽取35,774名和6,336名患者数据进行测试。Mixtral在63%的病程记录中成功检测到错误,其中17%存在单一词误分类,主要由医学术语引起。大模型能有效纠正多种错误。在不同错误率下,特征表示性能对低于10%的错误率具有容忍性,但超过10%后显著下降。结论:当错误率低于10%时模型表现良好,而达到或超过10%时性能急剧下滑。因此,使用数据前必须评估其质量;对于高错误率数据,必须采取纠错措施以保证模型可靠性与有效性。

原文摘要 · Abstract (English)

Background: Data collected in controlled settings typically results in high-quality datasets. However, in real-world applications, the quality of data collection is often compromised. It is well established that the quality of a dataset significantly impacts the performance of machine learning models. Methods: A rudimentary error rate metric was developed to evaluate textual dataset quality at the token level. Mixtral Large Language Model (LLM) was used to quantify and correct errors in low quality datasets. The study analyzed two healthcare datasets: the high-quality MIMIC-III public hospital dataset and a lower-quality private dataset from Australian aged care homes. Errors were systematically introduced into MIMIC at varying rates, while the ACH dataset quality was improved using the LLM. Results: For the sampled 35,774 and 6,336 patients from the MIMIC and ACH datasets respectively, we used Mixtral to introduce errors in MIMIC and correct errors in ACH. Mixtral correctly detected errors in 63% of progress notes, with 17% containing a single token misclassified due to medical terminology. LLMs demonstrated potential for improving progress note quality by addressing various errors. Under varying error rates, feature representation performance was tolerant to lower error rates (<10%) but declined significantly at higher rates. Conclusions: The study revealed that models performed relatively well on datasets with lower error rates (<10%), but their performance declined significantly as error rates increased (>=10%). Therefore, it is crucial to evaluate the quality of a dataset before utilizing it for machine learning tasks. For datasets with higher error rates, implementing corrective measures is essential to ensure the reliability and effectiveness of machine learning models.

数据质量医疗AI大模型错误检测

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。