Lung cancer has been responsible for most cancer deaths worldwide in 2020 and early identification is critical to improve outcomes. [1, 2] A hallmark of lung cancer is small lesions called nodules, which can be either benign or malignant. Distinguishing the two is important: suspected malignancy often requires a bronchoscopic biopsy, while it is desirable to avoid unnecessary biopsies, which is an invasive procedure associated with a risk of complications . To improve diagnosis of lung cancer and assessing malignancy, radiologists in training need to practice their diagnostic skills. However, practicing with a variety of cases can be challenging due to patient privacy, disease imbalance and reduced guidance from medical staff due to COVID-19. [4 – 6]
We have been working on a solution using Semantic Image Synthesis (SIS) networks to synthesize new and realistic 2D lung CT images. The SIS network requires a semantic label map as input and creates a realistic output image constrained by the semantic labels. By adjusting the labels in this semantic map, we can control the constraints of the output. However, the representation of the lung nodule is only a very small part of the 2D image, and some slices may not even include a nodule. This causes an imbalance of the different semantics represented on the map. We found that, at present, it is difficult to control the more subtle characteristics of the nodule to be synthesized in the image.
To solve this problem, we are planning to create a multimodal SIS network. The second step is to focus solely on nodule synthesis and its realistic integration into the synthesized 2D lung CT image.
The goal of the project is to create a multimodal semantic image synthesis network through which we can more realistically control nodule characteristics in a synthesized 2D lung CT image. Specifically, we seek to
- extend the current pre-processing pipeline to get the required inputs for a multi-modal SIS network
- create a multimodal SIS network that synthesizes 2D lung CT images and focuses on nodule characteristics
- design and apply an evaluation method to assess the realism of the synthesized nodules