AI-Powered training for the operating room of the future
From bottleneck to innovation
![[Translate to English:] Grata Operationssaal](/fileadmin/_processed_/5/5/csm_fortiss-projekt-grata-layout-3_64f7d72c21.jpg)
In recent years, robot-assisted surgery has evolved from a visionary technology into a clinical tool of growing relevance and is now considered a key medical technology. It enables minimally invasive procedures with unprecedented precision, reproducibility, and patient safety—especially in sensitive areas of microsurgery such as eye surgery. Yet this is precisely where a structural problem lies: the complexity of these systems is growing faster than the ability to train medical staff to use them efficiently and safely. This is particularly relevant given the increasing shortage of skilled personnel.
Traditional training methods reach their limits here. Actual operating room time is scarce, robotic systems are expensive, and training opportunities are limited and difficult to scale. At the same time, the need is growing: conditions such as age-related macular degeneration (AMD) are on the rise and require highly specialized procedures that demand extensive experience—skills that cannot be taught through theoretical knowledge alone. Training requires innovative methods and new approaches to teaching and practice.
The GRATA (Graph (RAG)-based Training and Education System) research project addresses this very issue using generative AI and a GraphRAG approach. RAG (Retrieval-Augmented Generation) is an approach in which generative AI not only generates answers based on learned knowledge but also accesses structured, interconnected knowledge sources to deliver context-dependent and factually sound results. The goal is to develop a fully digital, modular, AI-supported training and education system that maps the entire surgical workflow—using robot-assisted medical surgery, such as eye surgery, as an example—while taking both technical and organizational aspects into account. It is designed to support physicians and their assistants in all phases of complex procedures—from preparation to execution.
From static to adaptive learning
At the heart of GRATA lies a paradigm shift: moving away from a sequence of standardized instructional content toward a dynamic, data-driven process. The system integrates various technological components into a seamless learning environment that adapts to users’ individual progress.
A key element is the use of generative AI in the form of specialized medical language models. By training language models on medical terminology and medical knowledge, they are able to answer user questions, provide instructions, explain procedures, and engage in context-sensitive dialogues. It is crucial that the AI does not operate in isolation but is linked to structured knowledge models. This creates a robust foundation for transparent and medically accurate interactions.
In parallel, the system features advanced image processing and continuously processes sensor and robot data from the training environment in real time. The system records, analyzes, and compares movements, instrument positions, and process steps with stored target procedures.
Based on this, the system generates real-time feedback, detects deviations from the intended procedure, and supports medical trainees in refining their actions—for example, regarding the sequence of steps, the use of instruments, or adherence to clinical guidelines.
Semantics: Giving data meaning
A key aspect of the project is the structured representation of complex surgical processes. To this end, the State Research Institute of the Free State of Bavaria for Software-Intensive Systems (fortiss) is developing an intelligent database based on comprehensive semantic knowledge models. These models are crucial to ensuring that AI systems not only process data but can also understand relationships and respond appropriately to specific situations. The starting point is ontologies—structured knowledge models that establish a uniform vocabulary for surgical processes, the roles involved, and the instruments used. They make it possible to integrate different data sources into a consistent framework and make them usable for AI systems.
Building on this, process models are developed that map surgical procedures step by step. These models not only record what happens but also the context in which it occurs: Who performs which action, with which instrument, and under what conditions?
These models are supplemented by so-called scene graphs, which represent the entire surgical environment as a networked system. They link static information—such as devices or instruments—with dynamic states and interactions, thereby creating a comprehensive picture of the situation. This contextual information forms the basis for AI systems to understand, monitor, and provide targeted support during training scenarios.
fortiss’s GraphRAG approach combines this structured knowledge with generative AI (see figure on the left). Unlike traditional approaches, which are primarily based on unstructured text, it specifically draws on semantically modeled and interconnected domain knowledge. This allows medical contexts to be not only described but also interpreted in a context-appropriate, transparent, and reliable manner.
The path to intelligent assistance
Another advancement of GRATA lies in the intelligent integration of sensor data with knowledge-based models. Through the continuous collection of movement, instrument, and robot data, the surgical procedure is no longer merely documented but understood within its context.
As a result, the system automatically recognizes individual steps performed by medical staff and places them within the overall sequence of the procedure. For training, this provides significant added value: training progress is not only made visible but also explained, actively supported, and made easy to follow.
“Thanks to the knowledge models developed by fortiss, the system will ‘understand’ which phase a procedure is in, which steps have already been performed correctly, and where potential deviations may occur. These steps are to be identified in real time and addressed through targeted, situation-specific guidance. This creates a kind of digital mentor who not only observes the procedure but also provides expert guidance and helps improve it,” explains Dominik Mittel, a researcher at fortiss.
In practice, this means that the system can not only provide general information but also offer situation-specific answers based on the current step in the procedure. If a user—such as a trainee surgeon or anesthesiologist—asks, for example, about the next step, the appropriate instrument, or how to make a more precise incision, the system takes into account both the procedure to date and clinical guidelines, and uses this information to generate a precise, transparent recommendation. Furthermore, the system can explain the reasoning behind its suggestions and point out potential risks. The result is an assistance system that not only reacts but also actively supports the user through dialogue.
From simulation to reality
![[Translate to English:] Gruppenbild Grata](/fileadmin/_processed_/f/4/csm_fortiss-projekt-grata-layout-2_908eb6fb02.jpg)
In addition to semantic modeling and AI-supported interaction, simulation plays a central role. The goal is to create training environments that replicate real surgical situations as accurately as possible without putting patients at risk.
The integration of robotics into simulation systems transfers real-world movement sequences and system responses into virtual scenarios. This not only enables realistic training but also allows for the validation of new procedures and the optimization of surgical interventions under controlled conditions.
The combination of simulation and real-world data creates a development environment in which new approaches can be tested and improved iteratively. At the same time, the availability of training opportunities is significantly increased, as these can be utilized independently of actual operating room capacity, and the content can also be individually tailored to different levels of experience.
Neuer Standard für intelligentes Training
The first joint demonstration of the integrated system components at the Klinikum rechts der Isar of the Technical University of Munich brought together the contributions of the project partners and confirmed their functionality in a real-world environment. “This demonstration shows that the complex integration of AI, semantic models, and simulation is not only conceptually feasible but also practically implementable and can be consolidated into a single system,” emphasizes fortiss researcher Alexander Perzylo.
The new technology is intended not only to improve the quality of training but also to enhance the safety of surgical procedures. GRATA aims to establish a new standard for training in robot-assisted surgery, thereby exemplifying how modern medical technology is evolving: interdisciplinary, data-driven, robot-assisted, and AI-based. In the long term, the platform is therefore also intended to be applied to other medical disciplines.
“In the long term, we want to use this to establish a European standard: a validated platform that is flexible in its application, increases patient safety, and helps address the shortage of skilled professionals,” says Dr. Ali Nasseri, Professor of Medical Autonomy and Precision Robotics at the Technical University of Munich.
The combination of structured knowledge representation, generative AI, sensor-based process analysis, and simulation is fundamentally transforming surgical training. This bridges the gap between technological innovation and clinical application—and paves the way for a new generation of intelligent training systems in the operating room. As a result, the system has an impact not only on training but also on the further development of clinical processes themselves.
