DeepSeek-R1在双语眼科复杂推理中表现最优，超越多款顶尖大模型。
DeepSeek-R1 Outperforms Gemini 2.0 Pro, OpenAI o1, and o3-mini in Bilingual Complex Ophthalmology Reasoning
- 基于真实眼科考题构建双语测试集，评估模型推理能力。
- 中文准确率达86.2%，英文达80.8%，显著高于其他模型。
- 适合医学AI研发、临床辅助决策系统开发者参考。
目的：评估DeepSeek-R1及其他三款新发布大语言模型（LLMs）在双语复杂眼科病例中的准确率与推理能力。方法：从中国眼科高级职称考试中收集130道选择题（诊断类39题，管理类91题），分为六个主题，并用DeepSeek-R1翻译为英文。在2025年2月15日至20日间，以默认配置生成DeepSeek-R1、Gemini 2.0 Pro、OpenAI o1和o3-mini的响应。准确率定义为正确回答题数占比，漏答或多答均计为错误。推理能力通过分析推理逻辑及错误原因评估。结果：DeepSeek-R1整体准确率最高，中文题达0.862，英文题达0.808。Gemini 2.0 Pro、o1、o3-mini中文准确率分别为0.715、0.685、0.692（均P<0.001 vs DeepSeek-R1），英文分别为0.746（P=0.115）、0.723（P=0.027）、0.577（P<0.001）。DeepSeek-R1在六项主题中五项中文与英文均领先。其在中文管理题中表现亦显著更优（所有P<0.05）。推理分析显示四模型逻辑相似，主要错误包括忽略关键阳性病史、忽略阳性体征、误读医疗数据及过度激进。结论：DeepSeek-R1在双语复杂眼科推理任务中优于当前主流模型，虽临床应用仍存挑战，但具备辅助诊疗与决策支持潜力。
原文摘要 · Abstract (English)
Purpose: To evaluate the accuracy and reasoning ability of DeepSeek-R1 and three other recently released large language models (LLMs) in bilingual complex ophthalmology cases. Methods: A total of 130 multiple-choice questions (MCQs) related to diagnosis (n = 39) and management (n = 91) were collected from the Chinese ophthalmology senior professional title examination and categorized into six topics. These MCQs were translated into English using DeepSeek-R1. The responses of DeepSeek-R1, Gemini 2.0 Pro, OpenAI o1 and o3-mini were generated under default configurations between February 15 and February 20, 2025. Accuracy was calculated as the proportion of correctly answered questions, with omissions and extra answers considered incorrect. Reasoning ability was evaluated through analyzing reasoning logic and the causes of reasoning error. Results: DeepSeek-R1 demonstrated the highest overall accuracy, achieving 0.862 in Chinese MCQs and 0.808 in English MCQs. Gemini 2.0 Pro, OpenAI o1, and OpenAI o3-mini attained accuracies of 0.715, 0.685, and 0.692 in Chinese MCQs (all P<0.001 compared with DeepSeek-R1), and 0.746 (P=0.115), 0.723 (P=0.027), and 0.577 (P<0.001) in English MCQs, respectively. DeepSeek-R1 achieved the highest accuracy across five topics in both Chinese and English MCQs. It also excelled in management questions conducted in Chinese (all P<0.05). Reasoning ability analysis showed that the four LLMs shared similar reasoning logic. Ignoring key positive history, ignoring key positive signs, misinterpretation medical data, and too aggressive were the most common causes of reasoning errors. Conclusion: DeepSeek-R1 demonstrated superior performance in bilingual complex ophthalmology reasoning tasks than three other state-of-the-art LLMs. While its clinical applicability remains challenging, it shows promise for supporting diagnosis and clinical decision-making.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。