Project Description
Robotic ultrasound can improve the repeatability and automation of medical scanning. However, many existing robotic ultrasound systems still rely on fixed scan paths, such as raster or spiral scanning. These methods cannot dynamically adjust probe motion according to real-time ultrasound image quality. As a result, when the target structure is unclear, the anatomical boundary is off-centre, or some regions are insufficiently reconstructed, fixed-path scanning may reduce the accuracy of 3D reconstruction.
This project will build a simulated robotic ultrasound system for 3D forearm reconstruction based on the open-source SonoGym framework. The student will first reproduce the robotic ultrasound reconstruction baseline in SonoGym and then extend it from the original spine scenario to a forearm scenario, including structures such as the radius, ulna, and surrounding soft tissue. Based on this adapted simulation environment, the student will develop an ultrasound-based Vision-Language-Action policy, referred to as US-VLA, which enables the robotic ultrasound probe to predict the next scanning action based on real-time ultrasound images, current reconstruction status, and language instructions.
Unlike fixed-path scanning, the proposed system will adapt probe motion based on image feedback. For example, when the bone boundary is unclear, the system may adjust the probe angle; when the target structure is off-centre, the system may move laterally; when a region is insufficiently reconstructed, the system may actively rescan that area. Language instructions can be used to specify different task goals, such as “reconstruct the radius surface”, “reconstruct the ulna surface”, “scan both forearm bones”, or “rescan the low-confidence region”.
Existing robotic control and scanning code is already available for this project. Therefore, the student does not need to build a real robotic system from scratch. The main focus will be on implementing and evaluating an image-feedback-driven US-VLA scanning strategy in simulation, and comparing it with fixed scanning, coverage-guided scanning, and non-language-based policies. After the simulation experiments, if time and ethical approval allow, the system may be further tested on 3–6 healthy volunteers for small-scale technical validation of real forearm ultrasound scanning and 3D reconstruction.
Main Tasks
- Reproduce the robotic ultrasound reconstruction baseline in SonoGym.
- Extend the SonoGym reconstruction task to a simulated forearm anatomy.
- Design a forearm 3D reconstruction simulation workflow based on the existing robotic scanning code.
- Train a lightweight US-VLA policy that predicts the next probe motion based on ultrasound images and language-conditioned goals.
- Compare the proposed method with fixed scanning, coverage-guided scanning, and image-only policies.
- If conditions allow, perform a small-scale validation on 3–6 healthy volunteers.
Expected Outcomes
- A SonoGym-based robotic ultrasound simulation environment for forearm reconstruction;
- An ultrasound image-feedback adaptive scanning strategy;
- A lightweight US-VLA model;
- 3D reconstruction results of the radius and ulna;
- Small-scale technical validation results on 3–6 healthy volunteers, if approved and feasible;
- A paper-style technical report for robotic ultrasound or robot learning research.
Required Skills
This project is suitable for students interested in robotics, medical imaging, 3D reconstruction, deep learning, or vision-language models. Basic Python programming skills are required. Experience with PyTorch, robot coordinate transformations, image processing, point cloud processing, ROS, vision-language models, imitation learning, or reinforcement learning is beneficial but not mandatory.
Contact
Interested students can contact: dezhi.sun@utwente.nl