ASTRA: surgical training system for fetal surgery based on augmented reality and game technology

Interventions on fetuses can treat potentially very serious conditions for unborn babies and are among the most challenging to perform due to difficulty of access, reduced vision and manoeuvrability, a lack of information about what to expect once inside the patient, and the patient’s extreme fragility.

In recent years, to achieve a realistic simulation of the patient, major breakthroughs have been achieved in medical imaging techniques, AI and mixed reality. These advances are of paramount importance to surgeons as they enable them to plan surgery well in advance and train all of the most challenging steps of their interventions before entering the operating room.

Using the Unreal video game engine and Magic Leap augmented reality glasses, ASTRA is to develop fetal surgery simulation technology that will represent a significant breakthrough in training for this type of surgery

Creating a surgical training system based on augmented reality is the main goal of project ASTRA, thanks to funding by EPIC MegaGrants and the leadership of Mario Ceresa, a researcher with the BCN MedTech Group at the UPF Department of Information and Communication Technologies (DTIC) and Miguel A. Gonzalez Ballester, ICREA researcher and director of the BCN MedTech research unit. It will run for one year and enjoys the support of Elisenda Eixarch, an expert fetal surgeon and coordinator of the “Fetal neural development” and “Fetal therapy and surgery” lines at the BCNatal Fetal Medicine Research Center, affiliated to Hospital Clínic, Hospital de la Maternitat and Hospital Sant Joan de Déu.

“Thanks to the support of EPIC MegaGrants, we will develop our trainer using the Unreal Engine and Magic Leap augmented reality glasses. This will be a major breakthrough that could change the way training for fetal surgeries is carried out for ever and will have a positive impact on these patients’ living conditions”, reveals Mario Ceresa, principal investigator of the project. “The pioneering nature of ASTRA has been made possible thanks to the constant efforts by our research unit in fetal surgery in past years and projects such as FIRST and MIIFI”, comments Miguel A. González Ballester, director of the BCN MedTech unit.

Surgical training for twin-to-twin transfusion syndrome

The training system will focus on twin-to-twin transfusion syndrome (TTTS). “TTTS is a condition that occurs in monozygotic twin pregnancies where the fetuses are both connected to a single placenta. In one-third of cases, there is an imbalance in blood flow that can lead to the death of both twins. When a case of TTTS is detected, the only option is to separate the blood circulation of the two fetuses by coagulating the connections of blood vessels using laser”, comments the expert in fetal surgery, Elisenda Eixarch.

To create a useful environment for surgeons to train, the textures of all objects with which they interact will be as realistic as possible

ASTRA will proceed by obtaining recordings of live TTTS surgeries using RGB cameras and state-of-the-art deep learning networks to annotate and segment data of interest and the major events. “These data will form the basic training dataset for our application”, Ceresa points out.

Augmented reality user interfaces will be created for natural and comfortable fetal surgery training and simulation to enable the surgeon to explore patients’ anatomy and the steps involved in performing real TTTS surgery. To create a useful environment for surgeons to train, “we will ensure that the textures of all objects with which they interact are as realistic as possible and that the video streams simulated by our systems are comparable to those surgeons actually encounter in a real operation”, says Ceresa, principal investigator of the ASTRA project.

###