Periodic Reporting for period 5 - PULSE (Perception Ultrasound by Learning Sonographic Experience)
Reporting period: 2022-11-01 to 2023-04-30
To our knowledge this is the first body of work to attempt to bridge the gap between an ultrasound device and the user by employing a machine-learning solution that embeds clinical expert knowledge (through measuring perception and actions) to add interpretation power.
The innovation in PULSE is to apply the latest ideas from machine learning and computer vision to build, from real world training video data, computational models that describe how an expert sonographer performs a diagnostic study of a subject from multiple perceptual cues. Novel machine-learning based computational model designs were developed for different tasks (recognising standard planes, gaze-based and gaze-and-probe-based image and video guidance, describing sonographer actions, describing ultrasound images and video via text, describing sonographer skill, and summarising and characterising clinical workflow) based on probe and eye motion tracking, audio, image processing, and knowledge of how to interpret real-world clinical images and videos acquired to a standardised protocol. The underlying premise of our research is that by building models that more closely mimic how a human makes decisions from ultrasound images, considerably more efficient and powerful assistive interpretation methods can be built than have previously been possible from still US images and videos alone.
The overall objectives of the technical research were:
1. To develop a rich lexicon of sonographer words (vocabularies and languages) to describe US videos, the annotated datasets, and methods and software for accurately and reliably describing real world clinical ultrasound video content.
2. To build methods and software for describing ultrasound video content both for sonographer training and assistive technologies for clinical tasks.
3. To compare automatic description by using combined ultrasound video and probe motion information, and video, probe and eye motion information relative to ultrasound video alone.
The research underpins new multi-modal ultrasound imaging technology that may be developed further to have economic, healthcare and social benefits across Europe and beyond. The focus in the project was on feasibility demonstration. Software methodologies were developed and evaluated on real world obstetric US data in collaboration with clinical experts and trainees to validate the new approaches and to understand what the next translational steps might be towards potential future use in routine US scanning services in hospitals or the community.
The data was used to both study clinical sonography from a data science perspective for the first time as well as enable technical research on algorithms underpinning assistive tools for clinical sonography tasks which are informed by sonographer perceptions and actions.
In terms of dissemination, results have been presented as papers and keynotes at top academic international medical image analysis conferences as well as obstetrics and gynecology congresses. The work has received conference paper and workshop awards and appeared on the front pages of journals and conference highlights. The work on gaze prediction is being considered for commercial exploitation. The project has trained medical image analysis doctoral students and postdoctoral researchers and clinical fellows in healthcare AI.
The PULSE custom-built system allowed us to capture information about key perceptual cues – eye movement and probe motion - lost to conventional image- and video-based interpretation algorithms which only have the video stream of images to work with.
Using multi-modal analysis we studied the visual search strategies employed by full-qualified and trainee sonographers for instance.
We were also interested in questions such as whether trainees and full-qualified sonographers follow different visual search strategies, and whether there are different visual search strategies amongst experts.
Knowledge gleaned from these studies supported developments of assistive technologies to support sonography guidance and image reading/interpretation.
In summary, key outputs from PULSE were in the following areas:
1. Clinical sonography data science - greater understanding of clinical sonography workflow and sonographer skills/skills assessment.
2. Assistive technologies for interpreting ultrasound images - new machine-learning based models to assist in ultrasound standard plane detection and image interpretation.
3. Assistive technologies for ultrasound guidance - New machine-learning based multi-modal models to assist in ultrasound guidance for simple and complex tasks.
4. Video analysis - natural processing language: methodology to allow key information from hard to interpret ultrasound video to be communicated to a non-sonographer via a text-based description.