Last December, in a world first, orthopaedic surgeon Thomas Gregory, of Hospital Avicenne, in Paris, undertook transplant surgery of an artificial shoulder using a mixed reality headband. The surgery was broadcast live with a collaborative platform of mixed reality and interaction with remote doctors around the world
Dr Thomas Gregory, head of orthopaedic surgery and trauma in Hospital Avicenne, University Paris XIII, France, specialising in surgery of the hand and upper limb, conducted the operation and was assisted by the Evolutis and HoloPortal platforms and Microsoft HoloLens holographic computer. During the procedure, Gregory could see holograms – 3D modelling of anatomical pictures of his patient – projected in real time in the operating room. He also had access to patient medical data and interactive tutorials during the operation.
Gregory also exchanged during surgery via Skype with Professor Emery and Dr Reilly (Imperial College London), Professor Yan Lae Moon (Chosun University Hospital, South Korea), Dr Sledge (hospital Lafayette, Louisiana, USA) and Dr Armstrong (Pennsylvania, USA) and they could interact during surgery by proxy holograms. This was the first time ever that such a comprehensive use of immersive and collaborative aspects of this mixed reality device was done in an operating room, while maintaining optimum quality of holographic images.
The mixed reality makes living holographic experiences while linking to the outside world. HoloLens is a standalone computer that requires no cable. Worn like a helmet by the surgeon during the operation, it allows him to capture information from the environment in which it is thanks to its built-in microphones and sensors. Images can then be shared with others, such as remotely via Skype.
The surgeon keeps total freedom of movement and action in an enriched environment, and holograms information on request. Connected to HoloLens, the HoloPortal platform has been specifically developed by TeraRecon and Vizua and utilises the computing power of cloud Azure from Microsoft. It allows the surgeon to view real-time medical information from the patient that could be useful to him during the operation: his medical records, his biological assessments digitised 2D imaging (radiological native cuts). The surgeon can also directly access the scanners and MRI (without converting templates/formats). Evolutis has developed a platform allowing the surgeon to have access to the operative technique and to the patient-specific planning of the prosthesis position.
Since the pre-operative consultation to post-operative assessment, mixed reality improves the visualisation of patient medical images. It can be used at different stages of the patient’s support to:
Inform patients in consultation on its pathology and intervention program. The patient is shown his own care course and can take ownership of its operation;
Exchange in multidisciplinary discussions on specific pathological cases or difficult assignments. An immersive meeting allows medical and nursing teams to visualise a patient’s virtual twin in a meeting room;
Access patient medical information before and during the operation;
Discuss with the patient and his colleagues the surgery that has been done. The mixed reality can help train surgeons on the acquisition of new techniques and skills; and
With mixed reality, the surgeon has additional support in his medical practice. By facilitating interaction with information that may be helpful to him, it also helps to reduce the response time for better care for patients.
OPN talks to Thomas Gregory about his innovative surgery and what the future holds in regards the platform of mixed reality.
OPN: As a specialist orthopaedic surgeon, could you tell us more about your experience and training background in this field?
TG: I am currently chief of the orthopaedic department at Avicenne Teaching Hospital and associate Professor at the University Paris XIII. My orthopaedic department has a strong expertise in upper limb and sports surgery. My area of interest is the development of enhanced surgery in orthopaedic procedures with the use of digital technologies.
This interest led to the creation of the Moveo Foundation (www.fondation-moveo.fr) for which I am the founding member and serve as the president.
The Moveo Foundation supports research projects aiming to modernise and enhance medical performance at all levels of patient management, and to decrease the risk of human error from the medical profession, with the use of modern digital technology.
The Foundation’s goals are based on three observations:
Musculoskeletal diseases have the potential to affect 100 per cent of the population
Research on those diseases receive little financial support from public authorities
New technologies are revolutionising our daily lives and many medical disciplines, but orthopaedic surgery has had no major digital changes in the past several years.
With the MOVEO Foundation, I wish to transform surgical practices in orthopaedic surgery, particularly in shoulder and upper limb surgery, so they are more modern and connected, reflecting the image of a fast-changing society.
OPN: How did you decide to use mixed reality in your surgery?
TG: In June 2014, I performed the world’s first surgery using the medical application for the Occulus Rift virtual reality headset. Currently, I am running research projects focused on the use of related technologies, including:
Virtual reality (VR) to create a surgical simulator for training physicians
Augmented reality (AR) to improve the accuracy and security of minimally invasive surgery
Connected devices (CD) to gain a thorough understanding of a patient’s individual daily activity in order to develop patient-specific treatments.
OPN: Did you have to undertake a lot of specific training with the equipment before the surgery took place?
TG: The equipment was developed by a major US computer-based company. It is very intuitive. Of course, you need a couple of minutes to become familiar with the device and a couple minutes more to get familiar with the applications, but not more than that.
OPN: Tell us a little more about the live procedure in December? Any hurdles you had to overcome or difficulties beforehand or during the surgery?
TG: Positioning the scapula part of a shoulder arthroplasty is challenging. Firstly, the bone stock is limited, especially in the case on 5 December, because the patient was relatively short. Secondly, you have a limited access to the scapula bone.
You can only see the articular surface, which was deformed due to the erosion caused by the patient’s condition. The rest of the bone was covered by the thoracic soft tissues. Thirdly, the optimal position for the prosthesis is different for each individual and malalignment of the prosthesis typically causes negative outcomes. These difficulties were solved by coupling a clever Glenoid Positioning Guide we developed with Aventum Devices, with the mixed reality system.
With the headband, I was able to have access to the patient-specific prosthesis position we had planned before the surgery and to the whole operative technique developed in 3D holograms based on the planned position. Hence, while I was placing the prosthesis guided by the Glenoid Positioning Guide, I was able to compare, stage by stage, what I was doing with the appropriate placement. I was also able to drag the 3D reconstruction of the scapula, scaled 100 per cent, right before my eyes, with a simple gesture in front the headband. I was then able to move the image of the CT scapula with a manual adjustment so it was over the visible part of the bone.
I also shared what I saw in this headband with four other surgeons in the US, UK and South Korea, who were able to send information in the operator’s field of vision, via Skype, all throughout the intervention taking place in France. This is the first time ever that such a complete use of immersive and collaborative aspects of the technology have been implemented in an orthopaedic surgery of the shoulder.
OPN: How does this technology benefit the patient?
TG: Compared to a classic surgical procedure, the AR headset provides improved outcomes for both the surgeon and patient, without reducing the safety of the procedure. Indeed, thanks to 3D holograms, the keyframes of the operation as well as critical organs (nerves, arteries etc) are shown in real time to the surgeon.
This provides additional accuracy and safety in the procedure and reduces the operating time.
The safety of the patient, which all physicians must keep in mind, does not seem to be compromised by this new device. Indeed, according to our experience, no major adverse events occur during AR surgical procedures. On the contrary, the safety is probably improved because a surgeon can see the patient’s full anatomy, can receive additional input from other physicians, and can reduce the procedure time, which means a lower risk of surgical infection.
OPN: Can it be applied to all forms of surgical procedures?
TG: I believe so. To make a parallel with civil aviation, the headband is for the surgeon like a cockpit for a plane pilot.
My colleague and friend, John Sledge, who is based in Lafayette, USA, was interacting with me through Skype on 5 December. He is a spine and lower limb surgeon and already has a couple of ideas for developing other applications beneficial to his daily practice.
OPN: What do you think this means for the future of orthopaedic surgery, given the huge advancements in augmented reality?
TG: So far, the cost of the technologies hinders the spread of digital innovations such as robotics or navigation, in the field of orthopaedic surgery. However, this is not the case with AR technologies. As the hardware of the AR technology is supported by leading computer-based groups, the cost of the headband is affordable for most hospitals and practitioners. Specifically, the AR headband costs $3,000, as compared to prices ranging from $100,000 to $1m for navigation systems or robots.
To view the video of the surgery from December, visit https://www.youtube.com/watch?v=xUVMeib0qek&has_verified=1
