In microsurgical procedures, surgeons use micro-instruments under high magnifications to handle delicate tissues. These procedures require highly skilled attentional and motor control for planning and implementing eye-hand coordination strategies. Eye-hand coordination in surgery has mostly been studied in open, laparoscopic, and robot-assisted surgeries, as there are no available tools to perform automatic tool detection in microsurgery. We introduce and investigate a method for simultaneous detection and processing of micro-instruments and gaze during microsurgery. We train and evaluate a convolutional neural network for detecting 17 microsurgical tools with a dataset of 7500 frames from 20 videos of simulated and real surgical procedures. Model evaluations result in mean average precision at the 0.5 threshold of 89.5–91.4% for validation and 69.7–73.2% for testing over partially unseen surgical techniques.

For the first time in literature, we investigate the capability of Generative Adversarial Networks (GAN) for synthesizing realistic images of microsurgical procedures and augmenting training data for surgical tool detection. We employ videos from practice and intraoperative neurosurgical procedures to train and evaluate two recent GAN models that have shown promise in high-resolution image generation: StyleGAN2 with Adaptive Discriminator Augmentation and StyleGAN2 with Differential Augmentation. Models were trained with limited data for both conditional and unconditional image generation, where the conditional models generated images with and without surgical tools. Our results show that the unconditional models achieved FID scores between 6 and 25 units lower than the conditional models for the two practice datasets. The best performance (FID= 42.16 and 25.17) was achieved in the Go-around practice task and was comparable to the previous benchmark performance of StyleGAN2 with Differential Augmentation. Experts’ visual inspection showed that while synthetic images had faults that exposed their true origin to the human eye, a sizable portion of them included identifiable surgical instruments. Experiments with object detection showed that augmenting the training data with synthetic microsurgical data improved the mean average precision for detecting tool tips in practice microsurgery datasets by 3%. Future work will include improving the quality of image synthesis and investigating key visual cues in expert assessment of surgical scenes for applications in robust surgical tool detection, bimanual skill evaluation

Optic image-guidance systems enable minimally invasive (MIS) approaches in surgery. However, available MIS-techniques limits both ergonomics and field of view (FoV), which can be detrimental for anatomical awareness and safe manipulation with tissues. Contemporary navigation techniques (i.e. neuronavigation) support spatial awareness during surgery. However, these techniques require time-consuming instrumentation and lack real-time precision needed in soft-tissue surgery. In this work, we utilize operative microscopes FoV as an unobtrusive source to support MIS-navigation with micro-instrument tracking. The FoV instrument tracking has been investigated in laparoscopy, however, high magnification, selection of instruments and bimanually variant characteristics of microneurosurgery make the current computational approaches challenging to adopt. In this work, we investigate potentials of spectral