用多智能体框架实现医疗软件长周期任务自动化,提升复杂操作准确率。
CarePilot: A Multi-Agent Framework for Long-Horizon Computer Task Automation in Healthcare
- 基于演员-评论家架构,结合双记忆机制预测下一步操作。
- 在医疗工作流基准上超越现有模型15.26%(闭源)和3.38%(开源)。
- 适用于医疗自动化、智能助手研发人员,解决长流程决策难题。
多模态智能体流水线正在改变人机交互方式,实现复杂现实任务的高效自动化。然而,当前研究多聚焦于短周期或通用场景(如移动端或桌面端),对医疗等专业领域中的长周期自动化仍缺乏探索。为此,我们构建了CareFlow——一个高质量人工标注的基准数据集,涵盖医学标注工具、DICOM查看器、电子病历系统及实验室信息系统中的复杂长周期软件流程。在该基准上,现有视觉语言模型表现不佳,难以处理医疗场景下的长周期推理与多步交互。为此,我们提出CarePilot,一种基于演员-评论家范式的多智能体框架。演员通过工具定位与双记忆机制(长期与短期经验)从界面视觉信息与系统状态中预测下一步语义动作;评论家评估动作效果,更新记忆,并提供执行或修正反馈以优化流程。通过迭代式智能体仿真,演员在推理阶段获得更鲁棒、更具推理意识的动作预测能力。实验表明,CarePilot在自建基准及分布外数据集上分别超越强闭源与开源基线约15.26%与3.38%,达到当前最佳性能。
原文摘要 · Abstract (English)
Multimodal agentic pipelines are transforming human-computer interaction by enabling efficient and accessible automation of complex, real-world tasks. However, recent efforts have focused on short-horizon or general-purpose applications (e.g., mobile or desktop interfaces), leaving long-horizon automation for domain-specific systems, particularly in healthcare, largely unexplored. To address this, we introduce CareFlow, a high-quality human-annotated benchmark comprising complex, long-horizon software workflows across medical annotation tools, DICOM viewers, EHR systems, and laboratory information systems. On this benchmark, existing vision-language models (VLMs) perform poorly, struggling with long-horizon reasoning and multi-step interactions in medical contexts. To overcome this, we propose CarePilot, a multi-agent framework based on the actor-critic paradigm. The Actor integrates tool grounding with dual-memory mechanisms (long-term and short-term experience) to predict the next semantic action from the visual interface and system state. The Critic evaluates each action, updates memory based on observed effects, and either executes or provides corrective feedback to refine the workflow. Through iterative agentic simulation, the Actor learns to perform more robust and reasoning-aware predictions during inference. Our experiments show that CarePilot achieves state-of-the-art performance, outperforming strong closed-source and open-source multimodal baselines by approximately 15.26% and 3.38%, respectively, on our benchmark and out-of-distribution dataset.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。