Investigative study on virtual training enhancing full immersion and situational awareness for surgical and procedural training.

Back to List
Revolutionizing Surgery with XR+AI Telepresence
William Bon Koo1
Mohsen Rostami1,2
Adam Choe1
Joon Chung1,2
1 Veyond Metaverse Inc, San Francisco Bay, CA 94552, United States
2 Department of Aerospace Engineering, Toronto Metropolitan University, Toronto, ON, Canada

1. Abstract / Preface

The significant disparity in surgical care quality due to limited access to expert surgical training is a stark reality that affects approximately 5 billion people. It compromises the quality and safety of roughly 310 million surgeries performed annually worldwide. These alarming statistics underline a systemic issue in global healthcare - the geographical and logistical barriers to safe and high-quality surgical care. In response, some companies have offered telepresence technologies, which allow users to see and hear from a remote location through the aid of remote sensing devices and a first-person perspective. In this article, we introduce Veyond Connect™, an innovative 3D photorealistic immersive telepresence platform based on eXtended Reality (XR), digital twin, and Artificial Intelligence (AI) technologies.Veyond Connect™ has the potential to democratize surgical training by breaking down these geographical and logistical barriers, thereby facilitating expert-led surgical training and proctorship on a global scale.

2. Problem Statements

According to the Lancet Commission on Global Surgery (“LCGS”), the lack of access to safe surgery is a critical global problem affecting 5 billion people on the planet. Access is defined by LCGS as (1) the patient being able to reach hospital in a timely manner, (2) availability of the necessary resources in hospitals and medical staff, (3) resources being of sufficient quality and safety, and (4) being affordable. There is a need for capacity to perform an additional approximately 143 million surgeries every year. These alarming statistics underline systemic issues in global healthcare: major disparities between regions and countries and the overall lack of infrastructure and specialist staff raising the need for more surgery capacity and medical/surgical training.

3. Surgery training & operation challenge

3.1. Shortage of Specialists

Although operations/surgeries are an integral part of the medical field, recent deficiencies of specialists could lead to unexpected consequences. The AMA predicts a shortage of medical specialists by the next decade and a shortage of medical specialists means some patients face long wait times for care in the ER. And the problem is expected to get worse in many places in the coming years.  Based on theAmerican College of Surgeons, there is expected to be a shortage of15,800-30,200 in all surgical fields by 2034. Some factors include the high cost of medical education, with student loan debt being a major cause of this drop. Based on the Education Data Initiative, the average cost of medical school per year is $60,497 per year, which is an amount many cannot afford without a student loan. The shortage of specialists had already been a problem in SouthKorea with only 137 surgical trainees applying to surgery departments, resulting in 50 empty spots (The Korean Times). This could lead to deficienciesin health care in general [1-3].

Figure 1. There is a projected shortfall of 18 million health workers by 2030 [4].

3.2. High Costs

Though essential, the high and rising costs of different operations could lead to inaccessibility in everyday life. Based on America’s Debt Help Organization, the average cost of operations varies between $17,500 - $170,000 based on the operation. There are also add-ons, such as surgeon’s fees and anesthesia, which could increase the final bill drastically from the already hefty operation fee.With the average annual salary of a US citizen being around $60,000, it is reasonable to say it would be a huge blow financially (SoFi Learn). The costs are rising as well, with Medical and Surgical supply costs rising 6.5% each year from 2017-2021, based on Definitive Healthcare. Without insurance, the cost can be overwhelmingly high for the average person and can be detrimental financially [5-7].

3.3. Lack of Operating Rooms

With the rising population and demand for accessible healthcare, the lack of operating rooms could bar basic requirements. Although the US has plenty, others do not. According to WilliamRussell, Costa Rica has about 8.7 hospitals per million people, while Israel has 9.17. The shortage of operating rooms in these countries could result in over crowding in different hospitals, with many unable to get the help they need[8].

Figure 2. The shortage of operating rooms could become a huge problem [9].

3.4. Inequality across the Nation.

Due to the high costs of surgical operations, there will be a visible gap between those who can afford the operations, and those who cannot. Based on Sander's Estate, with the top 0.1 percent of the American population owning 20 percent of the nation's wealth, the wage gap has never been this wider. With surgeries costing up to $135,000 in the US based on CBS, it would be difficult for the average citizen to be able to afford these operations, whilst the people who are better off would be able to have more access than those who are not. This inequality also extends across different countries, impacting some more than others. Less developed countries often have substantial wage gaps compared to nations like the US. Additionally, many individuals in these countries can't afford insurance. As a result, they are unable to afford the necessary treatments. [10-11].

3.5. Limited by Physical Distances

Based on different states, the number of hospitals varies, with often some state shaving more than others. Based on the American Hospital Association, there are about 6,129 hospitals in the US. Each state has a completely different number of hospitals, however. States such as New York have about 186 hospitals despite their relatively smaller size, while bigger states like New Mexico have only37, and states like Delaware have 8. This could lead to people not being able to have access to hospitals due to the reasonable distances that are required when traveling to a hospital. Outside of the US, continents such as Africa have even limited access, with 29 percent of the population living 2 hours or more away from the nearest hospital. The difference between each country is also great, with countries such as South Sudan only having 25% of the population having access to the hospital in less than 2 hours (The Conversation) [12-14].

Figure 3. Inequality of healthcare is present worldwide in 2019, with countries with a lower Healthcare Access and Quality (HAQ) Index score having less accessibility [15].

3.6. Increasing Complexity

As lives are dependent on surgical operations, many complex operations require the help of machinery due to human errors. Operations such as Craniectomy require removing a part of the skull to relieve pressure on the brain, and if done wrong could cause further brain damage, requiring extreme precision and technique. Another operation is called Coronary Revascularization, which relies on working with arteries and can lead to detrimental results if any errors are made during the procedure [16]. Due to this, there are not many specialists able to succeed in these surgeries, due to the danger of human errors. Machinery’s precision and programming could solve this problem and could perform without the trouble of human errors. With machines able to figure out multiple outcomes and solutions in a millisecond, it would be much more precise with fewer errors when handling complex surgeries (Elicik) [16-17].

3.7. Limited Accessibility

Due to the high costs of surgical procedures, most people who cannot afford these prices may find surgery inaccessible based on the WorldBank, with 400 million people in the world not having access to healthcare services, the rising prices of surgeries could create a rise in inaccessibility in healthcare if it becomes the norm. With the average salary being $55,640 a year, based on Indeed, Career Guide, it would cause a huge financial blow if an operation is needed, due to operations costing up to 5 digits. Less developed countries have a hard time even affording health care, and operations could put them in a tricky situation [18-19].

3.8. Human associated Errors

With operations requiring extreme precision, even a minor human error could be life-threatening to the patient, especially in complex surgeries. According to PubMed, it has been proven that human error is the leading cause of failures during surgeries, rather than communications or the system itself. Due to this, if human error is thrown out, more operations would be successful, with a drastic drop in side effects [20].

4. Currently available solution.

4.1. Telepresence

Telepresence is a medium in which transducers such as video cameras and microphones substitute for the corresponding senses of the participant. The participant is able to see and hear with the aid of remote sensing devices in are mote location from the first-person point of view. The user can interact and affect the remote environment by their actions. Telepresence has a variety of methods in which it can be used. One of these methods is communication.Communication using telepresence makes users able to not only interact verbally with those around the world but also interact with them physically.Telepresence is applicable in many different contexts which can include manipulating sea vessels, dangerous chemical experiments, and even surgeries.Telepresence allows for a higher degree of precision and concentration during surgical operations. The main difference between telepresence and videoconference is the objective of each virtual device. Video conference is a tool used to only communicate with one far away, however, telepresence aims to simulate real meetings by making it seem like both parties are face to face as it merges two places into one. It could also remotely accomplish tasks. An example of telepresence would be using robot arms to have work done even when not physically there. This would make telepresence a much more advanced version of video conference. As a result, both are useful, however, each has its usefulness in different situations and circumstances [18-20].

Figure 4. A prime example of Telepresence, integrating virtual meetings to real life [21].

4.2. Companies Offering Telepresence

Proximie: Proximie is an application that aims to digitalize operation rooms. As many surgeons are unable to access different operation rooms, it is intended for surgeons to have access to patients without physically being in the location. As it uses telepresence, content management, and data in sights, it is aiming to precisely capture data, analyze different operations, and share data, all while not even being in the operating room. It is one of the biggest in the world and has been used in 50+ countries, 800+ hospitals, 16k+ users, and 40 medical organizations [22].

ImmersiveTouch: Immersive Touch allows surgeons to practice their skills virtually, by portraying virtual simulations using VR and AR. The company aims to give surgeons a realistic surgery experience without using a live patient or cadavers, which helps with improving one's technique. Personalized medicine is their key goal, focusing on simulating a patient's pathology in 3D, for information about the patient to not be lost [24].

Medivis: Medivis is a company that uses AR technology to plan out and improve the outcomes of surgery.It aims to provide 3D visualizations of a human’s anatomy during surgery, such as CT scans and MRI images, which improves precisi on and a high chance of success. It also helps with surgical planning, with surgeons being able to precisely simulate surgery before, reducing complications. As of 2023, there are 40+ hospitals and medical schools deployed, 500+ surgeries planned and rehearsed, and 10+ countries employed [25].

SurgicalTheater: Surgical Theater is a company that focuses on its product 360°-3DXR and uses VR to plan surgeries. It helps surgeons visualize a surgery beforehand by reconstructing a patient’s anatomy using CT scans or MRI images and simulating the operation using 3D visualization. It also allows different surgeons from far away to interact with the 3D model, able to practice with the model even at long distances. The company is currently located in Cleveland, Ohio [26].

Medinbox: Medinbox is a company that specializes in emphasizing collaboration in an operation room even with distance. Their main product, Medinbox, is an integrated camera system in an operating room that allows operators from anywhere to have live transmission of the surgery. The camera is able to be controlled and can capture different shots from anywhere. They were founded in 2010 and are located in France [27].

Rods&Cones: Rods&Cones is a company that specializes in surgery observation even when not physically present in the operating room. The company mainly produces smart glasses, which provide the person with a good view of the surgery without being there, letting them see what a surgeon sees. Communication is also present, providing a good way of transmitting information. They attempt to change healthcare for the better. The company is active in 85+ countries, with 1400+ hospitals using their product [28].

Augmetics: Augmetics is a company that aims to improve surgical outcomes using AR technologies. Their key technology is the xvision Spine System, which gives a full view of the patient’s anatomy to the surgeons and gives precise instructions on how to utilize the instruments and implants on a patient’s spine. With the technology being FDA cleared, it has been used to treat over4000 patients, and implanting over 20,000 pedicle screws in 21 US states, with xvision applications having a success rate of 97-100%. The company is currently located in Arlington Heights, Illinois [29].

5. Veyond Connect™: A Value Proposition

Veyond Connect™ emerges as a game-changer in the field of surgical training and care by offering an advanced XR telepresence platform that blends the power of AR, VR, MR, AI, and haptics. This technology revolutionizes traditional surgical procedures and training by creating an immersive, real-time collaborative environment that is not restricted by geographical boundaries.

The value propositions of Veyond Connect™ lie in its ability to:

  • Facilitate real-time collaboration and training, enabling proctors and operators to interact as if co-located.
  • Leverage frontier technologies to create precise virtual emulations of real-world surgical scenarios, enriched with AI capabilities for enhanced interaction and personalization.
  • Offer an unmatched telepresence using pioneering real-time 3D volumetric streaming technology, enhancing the reach and scalability of the platform.

Through these unique offerings, Veyond Connect™ stands at the forefront of revolutionizing surgical care, driving greater efficiency and democratizing access to surgical expertise on a global scale.

5.1. Case Study: XR Surgical Training Center

The XR Surgical Training Center is a prime example of Veyond Connect's potential. With our technology, the center conducts regular sessions with partners in Latin America, exemplifying how we are making high-quality surgical training accessible beyond geographical constraints.

This tangible illustration serves to underscore the transformative potential of Veyond Connect™ in democratizing surgical training.

Figure 12. Immersive XR Surgery Training Center in Latin America

5.2. Overcoming Challenges

Like any innovative technology, Veyond Connect™ faces its unique set of challenges. One such challenge is navigating the regulatory landscape, such as obtaining FDA approval, as we consider expanding the applications of our technology.

In the initial phase, our focus is primarily on education, training, and collaboration, areas where the impact of our technology is profound and regulatory hurdles are manageable. As we gain experience, gather data, and prove the value of our platform in these areas, we anticipate expanding into procedural applications, an expansion that will require rigorous testing and validation to meet regulatory standards.

To address these potential future challenges, we are committed to working closely with regulatory bodies. Our approach is two-fold. Firstly, we prioritize evidence-based decision making. We are in the process of collecting a robust set of data through our various initiatives, like the XR Surgical Training Center, which will be crucial in demonstrating the safety and efficacy of our technology. Secondly, we will actively foster collaborations with medical professionals globally to validate and refine our solutions.

Through our proactive and innovative approach in tackling these challenges, we are committed to maintaining Veyond Connect™ at the forefront of healthcare solutions, revolutionizing surgical care, and training in the process.

5.3. Remote Proctoring with Veyond ConnectTM

Fostering real-time collaboration between proctors and operators, Veyond Connect™simulates a shared physical presence in the operating room. Harnessing the potential of Artificial Intelligence (AI), dynamic digital twin technology, haptics, and XR, our platform creates an immersive, interactive, and incredibly realistic environment.This environment allows medical professionals globally to collaborate in realtime as if they were physically present in the same operating room.  This ability to provide unparalleled telepresence during surgical procedures and training scenarios sets Veyond Connect™ apart from its competitors.

Asa frontier technology incorporating AR, VR, MR, and haptics, Veyond Connect™transcends the conventional limits of VR applications. The hallmark of Veyond Connect™ lies not merely in being an XR node, but an influential force reshaping surgical procedures and training.

Veyond Connect'sTM pioneering real-time3D volumetric streaming technology manifests unmatched telepresence, enabling an unlimited number of global professionals to share a virtual space for surgical collaboration and training. This capability to accommodate numerous participants from any location drastically enhances the reach and scalability of Veyond Connect™.

This two-pronged functionality distinctly positions Veyond Connect™ against competitors such as Proximie and Augmedics. Proximie, constrained by its 2D, non-immersive interface, and Augmedics, limited by specific imaging views and outdated hardware, fail to match Veyond Connect'sTM extensive XR offerings. Furthermore, unlike Osso VR, which focuses solely on training, Veyond ConnectTM facilitates real-time interaction during live surgeries and training scenarios, enhancing the degree of collaboration. Veyond Connect'sTM digital twin technology, a leap forward from the competition, yields an accurate virtual emulation of real-world surgical scenarios. In tandem with actual surgical tools, it enables strikingly precise interactions during live surgeries and training sessions. A masterstroke of integrating haptic feedback technology adds another dimension of realism, replicating the tactile sensations experienced during physical surgeries.

Figure 12. Pioneering immersive communication surgery, the next-gentraining platform.

The standout features of Veyond ConnectTM are the ability to provide best-in-class or world-class surgical training without the need for physical travel. With Veyond Connect™, operators or trainees can seamlessly connect with proctors who maybe located far away, enabling them to receive the highest quality training and guidance remotely. This eliminates the limitations of traditional training methods and brings expertise directly to the operator's fingertips

Figure 13. Veyond ConnectTMprovides best-in-class or world-class surgical training without the need for physical travel.

Meticulously designed with a user-centric approach, Veyond Connect™ not only enhances user engagement and the overall learning experience through its intuitive design and interactive learning modules but also facilitates remote surgical collaboration and training, eliminating the need for physical travel. Veyond Connect™ democratizes access to surgical expertise, significantly enhancing the quality of surgical procedures and patient care in areas where surgical specialists may be scarce.

With the help of groundbreaking digital twin technology and the integration of haptic feedback, Veyond Connect™ creates accurate virtual replications of real-world surgicalscenarios, facilitating precise interactions and seamless workflow integration.This takes it a step further by overlaying instructions directly onto thepatient using annotation, ensuring a highly accurate and immersive experience.

Veyond ConnectTM utilizes AI to enhance the recognition, interpretation, and interaction with the digital twin. In addition to identifying and manipulating objects in the virtual space, it collaborates with the NaturalLanguage Processing (NLP) and Speech Recognition system to respond to voice-activated commands related to visual tasks such as zooming into specificare as, highlighting components, or retrieving specific data from the digital twin. Also, Veyond ConnectTM leveragesAI-powered services to provide real-time language translation for seamless international collaborations. In addition to translation, we incorporate voice recognition capabilities to interpret and execute verbal commands related to the digital twin's manipulation and information retrieval. The NLP system works in tandem with the computer vision system to ensure verbal commands translate into the appropriate visual actions on the digital twin. Veyond ConnectTM also employs available AI-based cyber security solutions to ensure the security and integrity of user data. Furthermore, Veyond ConnectTM uses machine learning for predictive analysis and decision support. Finally, we manage and derive insights from the vast amount of data generated by the platform is done by the help of AI and using AI Veyond ConnectTM implements a reinforcement learning system for personalized and efficient training experiences.

Figure 14. Veyond Connect™ creates accurate virtual replications of surgical instruments using digital twin technology

Veyond Connect™ is designed to be a transformative force in healthcare, aiming to significantly reduce overall costs while increasing efficiency and quality of surgical care.

a. Remote Collaboration and Training: By facilitating remote surgical collaboration and training, Veyond Connect™ eliminates the need for physical travel and associated expenses. It allows surgeons and healthcare professionals to seamlessly connect and collaborate from different locations, reducing travel costs for both patients and medical staff.

b. Enhanced Efficiency and Precision: Veyond Connect™platform improves the efficiency and precision of surgical procedures. This leads to reduced risks of complications and the need for follow-up interventions. By providing a highly immersive and realistic environment, it enhances surgical skills and decision-making, resulting in better patient outcomes and cost savings.

c. Democratizing Access to Surgical Expertise: Veyond Connect™ allows healthcare professionals to deliver high-quality care in are as where specialists may be scarce. This democratization of access eliminates the costs associated with patient transfers and referrals, providing timely and efficient care to patients locally.

Figure 15. Veyond Connect™ democratizes access to surgical expertise, significantly enhancing the quality of surgical procedures and patient care.

d. Minimizing Surgical Errors: With an estimated 350 million surgeries performed globally each year, minimizing surgical errors is crucial.Veyond Connect™ enhances surgical precision and reduces the risk of errors, leading to better patient outcomes and potential cost savings associated with reoperations or complications.

e. Meeting the Growing Surgery Demand: The global healthcare system faces the challenge of meeting the growing demand for surgeries, with an estimated need for an additional 143 million surgeries annually. Veyond Connect™addresses this need by increasing surgical capacity and facilitating high-quality training, contributing to a more efficient and equitable healthcare system.

By optimizing healthcare spending, Veyond Connect™ aims to contribute to the evolution of the $8.3 trillion global healthcare industry. As we prepare for a full-scale launch, we are continually refining our product to maximize its impact on reducing healthcare costs, ensuring that patients receive optimal care while optimizing resource allocation and driving cost-effectiveness.

Veyond Connect™ not only transcends geographical barriers, allowing worldwide access to high-quality surgical training and expertise, but also revolutionizes traditional training methods. It empowers operators and trainees to connect remotely with proctors, thereby receiving high-quality guidance without limitations. The geographical location should not hinder the advancement of surgical care, and Veyond Connect™ paves the way for inclusive, accessible, and top-notch surgical training globally.

6. Conclusion

The global need for accessible, high-quality surgical care and training is undeniable. To address this vital demand, we introduced Veyond Connect™, a ground-breaking eXtended Reality (XR) telepresence platform powered by cutting-edge artificial intelligence, digital twin, and haptic technologies.  This platform's immersive capabilities allow for real-time communication and collaboration between medical professionals, transcending geographical barriers. Our solution democratizes surgical training, enabling expert-led guidance and proctorship on a global scale.

Veyond Connect™ has the potential to significantly transform the land scape of global surgery, addressing critical challenges such as surgical skill shortage, disparity in medical training, and the need for more surgical capacity. By facilitating remote proctoring and surgical collaboration, our technology holds promise for alleviating the scarcity of specialist care in underserved regions, which could subsequently decrease morbidity and mortality rates.

As we continue to refine and expand our product, Veyond Connect™aims to redefine healthcare delivery, optimize resource allocation, and revolutionize surgical care. We are committed to leveraging digital connectivity and innovative solutions to meet the most pressing issues in healthcare today, forging a new path in medical training and patient care. This is the dawn of a new era, and Veyond Connect™ is at the forefront.

Stay tuned for our upcoming white papers as we delve deeper into various aspects of our technology, use cases, and broader impact on healthcare.

Thank you for joining us on this transformative journey, as west rive to shape the future of global surgical care with Veyond Connect™.

7. References

[1]. Lee Hyo-jin, “Surgery-related departments unpopular among medical students”,, [Accessed July 2023].

[2]. American college of Surgeons, “Surgical Workforce”,, [Accessed July 2023].

[3]. Melanie Hanson, “Average Cost of Medical School”, average cost of medical school is %2460%2C497 per year,a year for medical school, [Accessed July 2023].

[4] KidsOR, “There is a Global Surgical Crisis”, [Accessed July 2023].

[5]. Healthcare Insights, “Annual changes in hospital medical supply costs”, and surgical supply costs increase 6%25 on average&text=Between 2017 and 2021%2C average,up by 10%25 on average, [Accessed July 2023].

[6]. Jacqueline DeMarco, “Average US Salary by State”,, [Accessed July 2023].

[7]. Maureen Milliken, “Hospital and Surgery Costs”,, [Accessed July 2023].

[8]. William Cooper, “Which Countries Have The Worst Healthcare In The World In 2023?”, Costa Rica&text=Costa Rica has the lowest,8.7 for every million people, [Accessed July 2023].

[9]. Incision, ”Digital Guide to the Operating Room”,, [Accessed July 2023].

[10]. Jessica Learish, “Most expensive medical procedures in the U.S., ranked”,, [Accessed July 2023].

[11]. The Guardian, “The rich-poor gap in America is obscene. So let’s fix it – here’s how”,, [Accessed July 2023].

[12]. “Hospital Statistics by State”,, [Accessed July 2023].

[13]. American Hospital Association, “Fast Facts on U.S. Hospitals, 2023”,, [Accessed July 2023].

[14]. Paul Ouma, Emelda Okiro, “People across Africa have to travel far to get to a hospital. We worked out how far”,, [Accessed July 2023].

[15]. Lancet Glob Health, “Assessing performance of the Healthcare Access and Quality Index, overall and by select age groups, for 204 countries and territories, 1990–2019: a systematic analysis from the Global Burden of Disease Study 2019”, 2022,

[16]. “The 6 Most Complicated Medical Surgeries”,, [Accessed July 2023].

[17]. Eray Eliaçık, “Artificial Intelligence vs. Human Intelligence: Can a game-changing technology play the game?”,, [Accessed July 2023].

[18]. Sarah Cumberland, Melanie Mayhew, “New WHO and World Bank Group Report”, We Are-,New WHO and World Bank Group Report Shows that 400,Poverty because of Health Spending, [Accessed July 2023].

[19]. Jennifer Herrity, “Average Salary in the US (With Demographic Data)”,, [Accessed July 2023].

[20]. Peter J Fabri, José L Zayas-Castro, “Human error, not communication and systems, underlies surgical complications”,, [Accessed July 2023].

[18].  William R. Sherman, Alan B. Craig, “Telepresence”, 2003,, [Accessed July 2023].

[19]. “What is telehealth?”, of telehealth-,What does telehealth mean%3F,computer%2C tablet%2C or smartphone, [Accessed July 2023].

[20]. Natália Mrázová “What are Telepresence and Video Conferencing?”, video conferencing%2C you can,two locations into one place, [Accessed July 2023].

[21] TANDBERG Corporation, “An Introduction to Telepresence Videoconferencing”,, [Accessed July 2023].

[22] Proximie, “We’re creating a system change in healthcare by digitizing the world’s operating rooms”,, [Accessed July 2023].

[23] Proximie, “Digital operating room developer Proximie raises $80m”,, [Accessed July 2023].

[24] Immersivetouch,, [Accessed July 2023].

[25] Medivis,, [Accessed July 2023].

[26] Surgical Theater,, [Accessed July 2023].

[27] Medinbox,, [Accessed July 2023].

[28] Rods&Cones,, [Accessed July 2023].

[29] Augmedics,, [Accessed July 2023].

[30] Heba Soffar, “Telepresence robots uses, advantages & disadvantages”,, [Accessed July 2023].

[31] Collaboration Superpowers, “The Downside of Telepresence”,, [Accessed July 2023].

[32] RF and Wireless Vendors and Resources, “Advantages of Telepresence | disadvantages of Telepresence”, or disadvantages of Telepresence system&text=➨It requires extremely less,inspite of having same protocols, [Accessed July 2023].