用网页版AR实现无标记头动追踪,帮神经外科医生预演机器人穿刺手术。
Web-based Augmented Reality with Auto-Scaling and Real-Time Head Tracking towards Markerless Neurointerventional Preoperative Planning and Training of Head-mounted Robotic Needle Insertion
- 用MediaPipe和React Three Fiber实现人脸定位与实时3D虚拟器械投影。
- 自动按用户头型缩放模型,并在转动时保持精准追踪,误差小。
- 无需专用设备,多用户可同时在线协作,适合教学与术前规划。
神经外科手术要求极高的精度与全面的术前规划以确保患者最佳预后。尽管技术不断进步,仍缺乏直观、易用的工具来提升手术准备与医学教育水平。传统方法难以提供沉浸式体验以可视化复杂手术过程及关键神经血管结构,而现有高级解决方案往往成本高昂或需专用硬件。本研究提出一种新型无标记网页端增强现实(AR)应用,用于神经介入术前规划与教学。基于MediaPipe实现精准面部定位与分割,结合React Three Fiber进行沉浸式3D可视化,实时将虚拟2-RPS并联位移器(Skull-Bot)投影至用户面部,模拟高精度手术器械控制。核心功能包括导入头部解剖结构后自动按用户尺寸缩放,以及对齐后实现头动实时自动追踪。该网页架构支持多用户同步访问,促进术中协作,也允许医学生实时观摩手术过程。一项包含三名参与者的试点研究通过多种头部旋转测试评估了自动缩放与追踪性能。本研究为神经外科术前规划与教育提供了一种低成本、可及性强且支持协作的工具,有望提升手术效果并完善医学人才培养。源代码已公开于https://github.com/Hillllllllton/skullbot_web_ar。
原文摘要 · Abstract (English)
Neurosurgery requires exceptional precision and comprehensive preoperative planning to ensure optimal patient outcomes. Despite technological advancements, there remains a need for intuitive, accessible tools to enhance surgical preparation and medical education in this field. Traditional methods often lack the immersive experience necessary for surgeons to visualize complex procedures and critical neurovascular structures, while existing advanced solutions may be cost-prohibitive or require specialized hardware. This research presents a novel markerless web-based augmented reality (AR) application designed to address these challenges in neurointerventional preoperative planning and education. Utilizing MediaPipe for precise facial localization and segmentation, and React Three Fiber for immersive 3D visualization, the application offers an intuitive platform for complex preoperative procedures. A virtual 2-RPS parallel positioner or Skull-Bot model is projected onto the user's face in real-time, simulating surgical tool control with high precision. Key features include the ability to import and auto-scale head anatomy to the user's dimensions and real-time auto-tracking of head movements once aligned. The web-based nature enables simultaneous access by multiple users, facilitating collaboration during surgeries and allowing medical students to observe live procedures. A pilot study involving three participants evaluated the application's auto-scaling and auto-tracking capabilities through various head rotation exercises. This research contributes to the field by offering a cost-effective, accessible, and collaborative tool for improving neurosurgical planning and education, potentially leading to better surgical outcomes and more comprehensive training for medical professionals. The source code of our application is publicly available at https://github.com/Hillllllllton/skullbot_web_ar.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。