Perception Ultrasound by Learning Sonographic Experience (PULSE) explores how the latest ideas from machine learning (deep learning) can be combined with "big data" from clinical sonography to develop new methods and understanding to inform the development of a next generation of ultrasound imaging capabilities that make medical ultrasound more accessible to non-expert clinical professionals. A principal novel element (to our knowledge, unique in the world) is to record expert sonographer gaze, probe movements and the spoken word of sonographers while they scan and this means "artificial intelligence" -based models are built based on, not only the recorded ultrasound video, but as well human perceptual information (this is the human knowledge that goes into the model building process). In total we acquired 1400 full length clinical scans recorded with a purpose-built acquisition system.
To our knowledge this is the first body of work to attempt to bridge the gap between an ultrasound device and the user by employing a machine-learning solution that embeds clinical expert knowledge (through measuring perception and actions) to add interpretation power.
The innovation in PULSE is to apply the latest ideas from machine learning and computer vision to build, from real world training video data, computational models that describe how an expert sonographer performs a diagnostic study of a subject from multiple perceptual cues. Novel machine-learning based computational model designs were developed for different tasks (recognising standard planes, gaze-based and gaze-and-probe-based image and video guidance, describing sonographer actions, describing ultrasound images and video via text, describing sonographer skill, and summarising and characterising clinical workflow) based on probe and eye motion tracking, audio, image processing, and knowledge of how to interpret real-world clinical images and videos acquired to a standardised protocol. The underlying premise of our research is that by building models that more closely mimic how a human makes decisions from ultrasound images, considerably more efficient and powerful assistive interpretation methods can be built than have previously been possible from still US images and videos alone.
The overall objectives of the technical research were:
1. To develop a rich lexicon of sonographer words (vocabularies and languages) to describe US videos, the annotated datasets, and methods and software for accurately and reliably describing real world clinical ultrasound video content.
2. To build methods and software for describing ultrasound video content both for sonographer training and assistive technologies for clinical tasks.
3. To compare automatic description by using combined ultrasound video and probe motion information, and video, probe and eye motion information relative to ultrasound video alone.
The research underpins new multi-modal ultrasound imaging technology that may be developed further to have economic, healthcare and social benefits across Europe and beyond. The focus in the project was on feasibility demonstration. Software methodologies were developed and evaluated on real world obstetric US data in collaboration with clinical experts and trainees to validate the new approaches and to understand what the next translational steps might be towards potential future use in routine US scanning services in hospitals or the community.
