Evaluating predictions from PickET¶
Centroid-based evaluation¶
This evaluation computes the precision, recall and F1-score for the predicted centroids by comparing the shortest Euclidean distance between predicted centroids and ground truth centroids.
Along with these computations, N (where, N is number of predictions from PickET) number of random predictions are also made to get the random baseline.
These computations are made using the centroid assessment script which can be run as follows:
where assessment_params.yaml is a configurations file for the assessment. A detailed description of this file can be found in the example.
Compare all the workflows¶
All the workflows can be compared after computing the centroid based evaluation by running the comparison script as follows:
python plotting/compare_workflows.py <metric> <dataset_id> <output_dir> <path_to_centroid-based_evaluation_output/*.yaml>
where metric can be Precision,Recall, F1-score, etc. Essentially, it can be any of the keys from the evaluation output files.
Compute particle-wise recall¶
We also checked which particles were most likely to get picked in order to then understand the characteristics of a particle that made it more likely to be picked by PickET. This was done by running the particle-wise recall script as follows:
This script generates several .yaml files, each of which contain the average recall across all tomograms for each particle type as specified in the ground truth file for the workflow specified in the name of the file. This file can be better visualized as a scatter plot which can be generated by the following scripts:
1. For TomoTwin tomograms
2. For CZI tomograms
These scripts can be run as follows: 1. For TomoTwin tomograms:
python plotting/plot_particlewise_recall_tomotwin.py <particle-wise_recall_fpath> <particle_details_fname>
The particle_details_fname contains the details of each particle type in the dataset such as the molecular weight, radius of gyration, etc.
- for CZI tomograms:
Comparison with MiLoPYP¶
The details about the installation and running MiLoPYP are mentioned in the How I ran MiLoPYP document.
1. Convert the predictions to a .yaml format¶
The predictions obtained after running the first (Cellular content exploration) step were converted to a format similar to the particle centroids predicted by PickET (See also PickET outputs) using the conversion script. This script was run as follows:
python accessories/convert_milopyp_preds_to_yaml.py <output_from_MiLoPYP> <input_text_file_to_MiLoPYP> <output_dir>
where <output_from_MiLoPYP> is the all_output_info.npz generated by the inference step of the cellular content exploration module of MiLoPYP; <input_text_file_to_MiLoPYP> is the .txt provided as input to MiLoPYP; and <output_dir> is the path to the directory where the output .yaml files should be saved/
2. Compare the predictions from MiLoPYP and PickET¶
The precision, recall and F1-score on the MiLoPYP predictions can now be computed in the same way as that for PickET predictions (See evaluating PickET predictions).
The comparison can then be plotted by running the plotting script as follows:
where <path_to_MiLoPYP_evaluation> is the output .yaml file generated from running the evaluation step above on MiLoPYP predictions; similarly, the <path_to_PickET_evaluation> is the output .yaml file generated from running the evaluation step above on PickET predictions.
Plot particle-wise segmentations (Fig. 6 in the paper)¶
Run the following commands, one at a time, for each each of the rows in the figure:
python plotting/make_a_particle_wise_segmentation_image.py /data2/shreyas/mining_tomograms/working/s1_clean_results_picket_v4/czi_ds-10001_denoised/instance_segmentations/instance_segmentation_0_gabor_kmeans_watershed_segmentation.h5 450 550 ./
python plotting/make_a_particle_wise_segmentation_image.py /data2/shreyas/mining_tomograms/working/s1_clean_results_picket_v4/czi_ds-10301_denoised/instance_segmentations/instance_segmentation_0_gabor_kmeans_watershed_segmentation.h5 225 275 ./
python plotting/make_a_particle_wise_segmentation_image.py /data2/shreyas/mining_tomograms/working/s1_clean_results_picket_v4/czi_ds-10301_denoised/instance_segmentations/instance_segmentation_1_gabor_kmeans_watershed_segmentation.h5 225 275 ./
python plotting/make_a_particle_wise_segmentation_image.py /data2/shreyas/mining_tomograms/working/s1_clean_results_picket_v4/czi_ds-10301_denoised/instance_segmentations/instance_segmentation_3_gabor_kmeans_watershed_segmentation.h5 225 275 ./
python plotting/make_a_particle_wise_segmentation_image.py /data2/shreyas/mining_tomograms/working/s1_clean_results_picket_v4/czi_ds-10301_denoised/instance_segmentations/instance_segmentation_4_gabor_kmeans_watershed_segmentation.h5 225 275 ./
Each of these commands will load a Napari window with the input tomogram, semantic segmentation, instance segmentation and a particle-wise segmentation for the input tomogram. Save a screenshot of the tomogram alone, tomogram overlayed with instance segmentation and instance segmentation alone on the Z-slice that gets opened by default. Once you close the Napari window, a set of .mrc files will be saved, each of which contains a binary matrix of the shape of the input tomogram. Each of these .mrc files is a segmentation for each particle type. Load these in ChimeraX, set the thresholds to 0.999 and color them as per the key provided in the script. Set the step size to 1, graphics to soft mode and save the image using the save command.