The research carried out during this Reporting Period has evolved from single lines into an integrated effort leading to demonstrations of increasingly surgical relevance and we have achieved the following results:
1. Knowledge modelling and representation. We have clarified the distinction between the top-down approach (model derived from books and expertise) and the bottom-up approach (model derived from data). In particular:
a. Top-down approach. We made progress in the development of methods to extract declarative and procedural knowledge of surgical procedures from robotic surgery textbooks. We are also designing methods on how to interface the description of a procedure extracted from textbooks written in Natural Language to the Answer Set Programming (ASP) framework used to plan the experiments demonstrated in this period.
b. Bottom-up approach. The approach followed in this research line has been to assume that each surgical procedure can be represented by a hybrid dynamic system consisting of several discrete states each corresponding to a specific action, e.g. approach, cutting, lifting, etc., and by a dynamic model representing the motions within the discrete state. With these assumptions, the data modelling consists of identifying the temporal sequences corresponding to each discrete state of the hybrid model, and to model the continuous motion within each state. This research led to the successful conclusion of Michele Ginesi doctoral work.
c. Simulation allows testing the task plan and the assumptions on the biomechanical properties of the tissues involved in the intervention. We are using simulation in different ways: to test a procedure plan, to keep an updated image of the anatomical environment, and as the background knowledge against which we compare the results of the real actions. Since reality is always right, discrepancies between the simulation and the data acquired from the field trigger a revision of the model to make it consistent with the anatomical behavior. To simulate in real time the anatomical behavior we use a Deep Neural Network (DNN), called BA-Net, which is trained with the synthetic data generated by the FEM off-line simulator. This research led to the successful conclusion of Eleonora Tagliabue doctoral work.
2. Planning. Thanks to the interaction between perception and planning, the plan variables (fluents) are instantiated at run-time using the sensor feedback, and when the nominal execution encounters an unexpected event, the fluents’ values are updated, new logic rules are triggered, and the plan sequence is modified to achieve task completion. This research led to the successful conclusion of Daniele Meli doctoral work.
3. Execution. Demonstrations and experiments are carried out with the DaVinci Research Kit (dVRK) of the robotics laboratory, although the system starts showing operational problems and random failures that affect its availability. The dVRK is based on the Standard (Classic) model of the da Vinci Surgical System, and it has been operational for about 20 years.
4. Monitoring. Perception in robotic surgery is key to successfully complete a task. Wehave expanded the sensor suite developed in the 2nd Reporting Period, with new sensors and algorithms to extract geometrical and physical properties of the anatomical environment. The sensor to measure the electrical impedance of the tissues have been integrated with an exploration algorithm that uses two instruments, instead of the original one. The measurement are calibrated with the approach developed in the previous Reporting Period, However, in a surgical environment, the anatomy is not steady and therefore we are developing the so called “anatomical SLAM”, which includes a semantic map, to distinguish between steady and moving organs. A new effort has started towards the development of force sensors for surgical instruments and to the identification of the forces involved in a surgical procedure. Echography images to enhance the perceptionare processed with a new method based on a Deep Neural Net, called PROST-Net.
5. Verification. After the successful completion of the Peg and Ring and of the Tissue Lifting tasks, we are now focusing on improving these technologies to make them suitable for the simulation of a kidney intervention.
