About - Authors - Project structure - Getting started - How To Run Experiments - Input and Output - Credits - License

This repository contains a deep learning project developed in collaboration with Gustave Roussy Hospital, within Machine Learning course at EPFL.

Correctly identifying tumor lesions on Computed Tomography (CT) scans is a crucial task in cancer diagnosis and treatment. Developing and improving deep learning models to automate tumor segmentation, thereby supporting radiologists’ work, could significantly reduce diagnostic errors and the time required to obtain a diagnosis. In this work, we propose a comprehensive pipeline for processing language–vision medical datasets and a 3D vision–language segmentation model comprising of a Language-Guided vision encoder with a Scale-aware Vision–Language Attention module, a Full-scale decoder, and a loss combining segmentation errors and contrastive errors. We compare and validate our approach on a model previously built at Gustave Roussy on abdominal CT scans with liver cancer lesions. Please cite the repo if you use our code :)

• Sofia Sannino
• Neila Benlamri
• Bipasha Goyal

Back to top

We provide Hydra-based execution scripts (with corresponding configs) to reproduce the full pipeline described in the report: preprocessing a dataset of CT volumes (NIfTI), segmentation masks (NIfTI), and text prompts (JSON), training the model, and running inference on new samples. The implementation is available in src and is documented throughout the codebase.

1. src/configs contains hydra configurations.
2. src/model contains the blocks of our network and the main block LSMS_model.py
3. src/datasets contains functions and main classes to proprocess the dataset and the main dataloader classes
4. src/loss contains our losses classes
5. src/infer contains helper functions to segment lesions on new CT scans
6. preprocess.py is the main file to preprocess the dataset.
7. train_1.py is the main file to train on the dataset with our proposed model.
8. infer_1.py is the main file to infer on a new CT scan.
9. train_old_model.py is the main file to train on the dataset with the baseline model.

Back to top

Python version: Python >= 3.9, <= 3.10 (tested on 3.10)
0. (Optional) Create and activate new environment using conda or venv (+pyenv).

a. conda version:

# create env
conda create -n project_env python=PYTHON_VERSION

# activate env
conda activate project_env

b. venv version:

# create env
python -m venv project_env

# activate env
source project_env/bin/activate

1. Install all required packages to run

   pip install -r requirements.txt

2. (Optional) Install pre-commit:

   pre-commit install

3. Follow the instructions to extract the dataset as .tar archive detailed in Input and Output. Back to top
   

Run the experiments after checking Input and Output.

1. To preprocess dataset with the report configuration

python preprocess.py 

2. To train experiment 1 with our proposed model

python train_1.py dataset._target_=src.datasets.MedVLSet #(To modify configs of new model in general go to `baseline.yaml`) 

3. To train experiment 1 with the baseline model

python train_old_model.py dataset._target_=src.datasets.MedVLSet #(To modify configs of old model in general go to `baseline_old.yaml`) 

4. To train experiment 2 with our proposed model

python train_1.py dataset._target_=src.datasets.MedVLSet2

5. To train experiment 2 with the baseline model

python train_old_model.py dataset._target_=src.datasets.MedVLSet2

6. To infer on a new CT scan:

• Inference config (infer.yaml): set ct (preprocessed CT path), features (training features path), model (weights path), and out (output file path). Provide the patient text in prompt and choose model_type (LSMS = proposed, LSMSLite = baseline).

• Training config (baseline.yaml): set patches_per_patient to 1 (single patch per patient) or 8 (multiple patches per patient).

  python infer_1.py

Back to top

Dataset
We preprocessed and trained on a subsample of liver cancer patients extracted from the public dataset AbdomenAtlas 3.0.

You can extract the already preprocessed sample from this drive, together with lsms_liver_prompts.json.

After downloading locally lsms_liver_subset.tar and lsms_liver_prompts.json, follow the instructions below.

Windows 10/11

1. Open PowerShell or Command Prompt

2. Go to the directory containing the archive:

   cd D:\path\to\archive

3. Extract:

   tar -xf dataset.tar

4. Write in src/configs/dataloader/preprocessing.yaml and src/configs/datasets/preprocessing.yaml, in root and in prompts the directory where you extracted lsms_liver_subset.tar and downloaded lsms_liver_prompts.json.

WSL

1. Extract:

   cd /mnt/d/path/to/archive
   tar -xf dataset.tar

2. Write in src/configs/dataloader/preprocessing.yaml and src/configs/datasets/preprocessing.yaml, in root and in prompts the directory where you extracted lsms_liver_subset.tar and downloaded lsms_liver_prompts.json.

macOS

1. Open Terminal

2. macOS ships with BSD tar:

   cd /Volumes/DriveName/path/to/archive
   tar -xf dataset.tar

3. Write in src/configs/dataloader/preprocessing.yaml and src/configs/datasets/preprocessing.yaml, in root and in prompts the directory where you extracted lsms_liver_subset.tar and downloaded lsms_liver_prompts.json.

Experiments output
The outputs of the experiments described in the report can be found in drive as well.

1. best_model folder contains the output of experiment 2 with our proposed model.
2. old_model_with_patches folder contains the output of experiment 2 with the Gustave Roussy model.
3. new_model_without_patches folder contains the output of experiment 1 with our proposed model.
4. old_model_without_patches folder contains the output of experiment 1 with the Gustave Roussy model.

Back to top

This repository is based on a heavily modified fork of pytorch-template and asr_project_template repositories. Back to top

Back to top