This repository contains the code to prepare the ADNI dataset for the experiments. It creates two different datasets, one used for a diagnostic task and the other for a prognostic task. Each of these is divided into 4 different modalities, namely assessment, biospecimen, image_analysis and subject_characteristics. The labels are reported in the annotations_<specs> files. For each modality it creates the csv file containing the related information and a yaml file containing the types of each column.

We used Python 3.9 for the development of the code. To install the required packages, it is sufficient to run the following command:

pip install -r requirements.txt

To prepare one of the two datasets, the diagnostic and the prognostic one, you have to change the config_name reported above the main function in the dataset_preparation.py file to match the related configuration file in the confs folder.

The two configuration files reported in the confs folder, namely diagnosis_preparation.yaml and prognosis_preparation.yaml, define all the necessary information to create the related datasets.

Starting from the first lines they define:

• the task, diagnosis or prognosis;
• the configuration file, reported in the paths folder, to be used to define the data_path (where the ADNI raw files are stored) and the save_path (where the created datasets will be stored).

task: diagnosis # or prognosis

defaults:
  - _self_
  - paths: ccaruso # or afrancesconi

Then for each modality there is a list of configuration files to be used for reading the related documents, but let us see an example for clarity.

The line below defines a configuration file to be used to load some information related to the assessment modality and stored in the ADASSCORES.csv file. Please note how the line is structured: diagnosis/assessment allows the configuration file to go to the folder of configuration files related to the diagnosis-related assessment (i.e., diagnosis/assessment folder), whereas the part after the @ symbol, i.e., modalities.assessment, will be used by the python code to build the modality. Note that the .0 at the end of the line is only needed to construct a dictionary, but the value written is irrelevant, it is sufficient that it is different from that of the other files used to construct the same modality.

  - diagnosis/assessment@modalities.assessment.0: adasscores
    
  # example of other files to be used to build the same modality
  - diagnosis/assessment@modalities.assessment.bla: adnimerge

Now let us see the content of a configuration file used to load some information related to a modality. As an example let us see the adasscores.yaml file.

filename: ADASSCORES.csv

pandas_load_kwargs:
  na_values: [-4]
  dtype: str # to avoid any conversion of the columns

filters:
  VISCODE: [bl]

columns:
  RID: id
  Q4:  int

The file defines:

• the filename of the file (reported in the data_path folder) to load;
• the pandas_load_kwargs, which are the pandas kwargs to be used to load the file;
• the filters to be applied to the file, which are a list of columns and the related values to be kept;
• the columns to be read from that specific file and to be inserted in the modality.

The latter is a dictionary, where the key is the name of the column in the dataset and the value is the type of the column, which will be used to construct the yaml file related to the modality. More specifically, the type can be one of the following:

• id: the column used as an identifier and utilized to join the columns of the different files related to the modality;
• float: the column is a float;
• int: the column is an integer;
• category: the column is a category (more generically a string).

The last part of the main configuration files (diagnosis_preparation.yaml or prognosis_preparation.yaml) is needed just to disable the automatic creation of an output folder and a log file that the hydra library creates by default.

  #############################################################
  - disable_hydra_outputs: disable_hydra_outputs # DO NOT CHANGE
  - override hydra/hydra_logging: disabled       # DO NOT CHANGE
  - override hydra/job_logging: disabled         # DO NOT CHANGE