AI医生在视频问诊中达到专家水平,可实时理解语音与影像信息。
Towards Expert-level Medical AI for Real-time Video Consultations

- 基于Gemini的多智能体系统,融合对话、推理与实时视听感知。
- 在100个临床场景中,表现不逊于30名全科医生的问诊与诊断能力。
- 适合医疗AI研发者及远程诊疗系统设计者参考。
语音-视觉交互是医患问诊的标准形式,能通过非语言线索有效评估病情。尽管文本型AI已有进展,但丢失了关键感知维度,对无法书面描述症状的患者不友好。早期尝试扩展医疗AI至视听交互已证明可行性,但未达临床医生水平。本文首次展示在实时视频问诊中达到专家级表现的AI系统——基于Gemini的多智能体系统AMIE(Video),其整合低延迟对话、临床推理与实时音视频感知。为指导开发,我们建立了远程医疗中临床视听线索的分类体系与自动化评估方法。在包含30名全科医生、15名患者演员和100个临床情景的随机客观结构化临床考试(OSCE)中,临床评估者认为AMIE(Video)在病史采集、诊断、治疗和体格检查方面表现与医生相当甚至更优;患者演员更偏好其对病情的解释与评估方式;而医生在建立医患关系与合作方面仍具优势。模态消融实验显示,患者更倾向使用视频界面而非纯文本聊天,因其沟通效率更高、更便捷且感觉被理解。局限性在于精细解剖定位、细微情感变化及高频运动捕捉。虽需进一步研究才能实现真实世界应用,但本成果标志着迈向能应对临床实践感官复杂性的AI辅助医疗的重要里程碑。
原文摘要 · Abstract (English)
Audio-visual interaction is the standard for patient-physician consultations, enabling natural communication and effective assessment of illness through non-verbal cues. While text-based AI has shown promise, it discards essential perceptual dimensions and limits patients who cannot articulate symptoms in writing. Early efforts to extend medical AI to audio-visual interaction have demonstrated feasibility but not reached clinician-level performance. Here, we provide the first demonstration of expert-level AI in real-time clinical video consultations using AMIE (Articulate Medical Intelligence Explorer) in a video configuration. AMIE (Video) is a Gemini-based multi-agent system integrating low-latency dialogue, clinical reasoning, and real-time audio-visual perception. To guide development, we established a taxonomy and automated evaluations for clinical audio-visual cues in telehealth settings. In a randomized Objective Structured Clinical Examination (OSCE) study with 30 primary care physicians (PCPs), 15 patient actors and 100 clinical scenarios, we compared AMIE (Video), its text-only counterpart AMIE (Text), and PCPs consulting via video. Clinical evaluators rated AMIE (Video) on par or better than PCPs in history-taking, diagnosis, management, and physical observation and examination. Patient actors preferred AMIE's approach to assessing and explaining conditions, while PCPs were preferred for rapport and partnership building. In modality ablation, patient actors preferred AMIE (Video)'s interface over text chat for communicative effectiveness, convenience, and feeling understood. Limitations remain in fine anatomical precision, subtle affective nuances, and high-frequency movements. While further research is needed before real-world translation, these results mark an important milestone toward AI systems capable of augmenting care across the sensory complexity of clinical practice.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。