Robotic ultrasound systems rely on consistent, repeatable scanning trajectories, but current methods for specifying these require computer interaction and predetermined force settings, making them unintuitive to operate. A prior fingertip-tracking pipeline can record a natural scanning demonstration, but existing force estimation networks are not accurate enough for a finger sliding across the forearm, so both the path and the contact force should instead be demonstrated intuitively through natural finger movement. Once acquired, ultrasound images must be segmented and reconstructed in 3D, but raw frames contain more information than needed, making it hard to isolate relevant regions. Gaze data from a HoloLens 2's eye-tracking could be converted into a spatial attention map indicating which bone the clinician intends to reconstruct, guiding segmentation and suppressing irrelevant structures. Since the final system should run without a nearby screen, the live ultrasound feed and growing 3D reconstruction must also be streamed and displayed within the HoloLens 2 at the correct anatomical position.
This thesis proposes such a human-in-the-loop system, delivering:
• an extended path-planning pipeline converting demonstrated fingertip paths into robot instructions with discretised contact-force levels;
• real-time AR visualisation of the live ultrasound feed and 3D reconstruction in the HoloLens 2;
• gaze-guided segmentation using eye-tracking-derived attention maps.
