Head-to-head comparison against NORDIC MATLAB implementation?

[tsalo]

Have you had a chance to a do a direct comparison to https://github.com/SteenMoeller/NORDIC_Raw?

[tsalo]

On a related note (probably not worth a separate issue), would you be able/willing to directly connect the parameters in patch-denoising's version of NORDIC to the parameters in NORDIC_Raw?

[paquiteau]

Hi! Yes we did, and we published a paper about it at ISBI 2023: https://hal.science/hal-03895194

The code for reproducing the article lies here: https://github.com/paquiteau/retino-pypeline/

My implementation of NORDIC was probably not good enough, I was not able to reproduced the same results (and I was not able to achieve the same code quality in python /s).

For the mapping of parameters, it should be straightforward; I will do it soon.

[tsalo]

Here are few ideas I had for a head-to-head comparison:

1. Magnitude without noise volumes
2. Magnitude + phase without noise volumes
3. Magnitude with noise volumes
4. Magnitude + phase with noise volumes

I think even running on one run/echo from https://openneuro.org/datasets/ds005250 would be enough for a test.

I assume it would be best to compare each output, but I don't actually have a great idea for how to compare the denoised data from patch-denoise's NORDIC to NORDIC_Raw's NORDIC. Maybe something simple, like making sure the denoised data are more similar to each other than to the un-denoised data? Oh, what about comparing them against MP-PCA denoised data as well? NORDIC_Raw denoised data should be more similar to patch-denoise NORDIC-denoised data than to patch-denoise MP-PCA-denoised data.

Also RMSD between the two g-factor maps. I think the g-factor map is deterministic, so it should be easy to do a direct comparison.

[paquiteau]

I think the comparison could be done on the downstream analysis (where each denoised data undergoes the same t/f-test for instance). Another solution is to look for correlations in times series between the two denoised versions (They should be fairly similar if everything goes well).

The dataset you linked is interesting, but at 2mm-iso (volume size of 110x110x72) with 238 volumes for the first series, I am not sure we would be in a good setup (Noise should be mostly thermal). A fully simulated fMRI run could be more convenient (and we would have more control on the parameters).

Otherwise, I could (probably) give you access to the data we used in our ISBI paper (6 volunteers scanned at 7T with EPI3D, 1mm-iso and TR=2.4s)

[tsalo]

When it comes to regular testing, I prefer to keep the test closer to the actual process, so I like the idea of correlating the denoised time series over running full analyses. From a "this is a useful denoising step" perspective, running the full analyses makes the most sense, but from a "this software works" perspective, just showing that the denoised data look right would be better IMHO.

I agree that a fully simulated run would be better for most of the tests, though testing on real data is also crucial since the assumptions in the simulated data may not always match what happens in real data.

FWIW, I agree that NORDIC probably doesn't do much at 3T (based on previous conversations with Logan Dowdle), but the protocol we used in ds005250 was taken from UMN/WashU collaborators, who have been publishing on how it's super useful (e.g., Moser et al. (preprint)), so... 🤷

[tsalo]

I figure this is a good place to enumerate differences I've noticed between patch-denoising and NORDIC:

1. It doesn't look like g-factor correction is done in patch-denoising.
   • I think this could be handled by the user by running an MP-PCA denoiser, grabbing the noise map, and scaling the input data based on that before running the NORDIC denoiser. Could be formalized as an example workflow for users rather than its own class or function.
2. NORDIC calculates what I think is a filtered phase map that it uses to correct the input data.
3. NORDIC fills in zero voxels (identified in the g-factor map) in the input data with random values.
4. NORDIC includes noise volumes for all steps. I'm not sure if this is a bug on NORDIC's side or not.
5. The following outputs from NORDIC would be useful:
   • g-factor map
   • SNR map
   • energy removed map (not sure how useful this one is)
6. patch-denoising doesn't seem to have equivalents of the following NORDIC parameters:
   • full_dynamic_range
   • phase_slice_average_for_kspace_centering
   • phase_filter_width
   • factor_error: could just be set by the user in noise_std I think
   • llr_scale: could just be set by the user in noise_std I think
   • patch_average: I don't actually know what this variable does in NORDIC. It's not the same as patch_avg, which matches up with recombination in patch-denoising.
7. NORDIC doesn't have equivalents to the following patch-denoising parameters:
   • mask
   • mask_threshold
   • n_iter_threshold (hardcoded as 10 in NORDIC)
8. Differences in parameters:
   • kernel_size_pca == patch_shape
     • NORDIC runs both the g-factor estimation and the PCA denoising, so there are separate parameters for the g-factor and PCA patch size and overlap.
   • patch_overlap_pca == patch_overlap
   • scale_patches + patch_avg (not a function parameter- just hardcoded variable) == recombination
     • scale_patches = True + patch_avg = True --> recombination = weighted?
     • scale_patches = False + patch_avg = True --> recombination = average
     • scale_patches = False + patch_avg = False --> recombination = center
     • scale_patches = True + patch_avg = False --> ???

[paquiteau]

Thanks for the detailled comparison ! Indeed g-factor has always been missing in patch-denoise (I lacked the time to get a proper implementation, note that there is now ggrappa which would give it to us (and very fast). Doing MP-PCA then NORDIC seems counter-productive to me, I don't think there is some benefit to run the methods twice. (The boost from MP-PCA would be significant enough already).

I will have a look at the NORDIC code again, but from a pure methodological aspect, I think that optimal denoising (optimal-fro, -nuc, -op) are better.

To your (very) detailled list I would add:

9. The masking to only compute patches that are in a given ROI.

[tsalo]

note that there is now ggrappa which would give it to us (and very fast).

The ggrappa documentation isn't very clear to me. Does it require k-space data or can you use it on already-reconstructed data?

Doing MP-PCA then NORDIC seems counter-productive to me, I don't think there is some benefit to run the methods twice.

I think the g-factor map is just the noise map from running an MP-PCA denoising. Then you scale the data by the g-factor map, and then pass the scaled data to the NORDIC denoising step. There are a few filtering/scaling things that happen differently between the two steps (g-factor estimation and NORDIC denoising), so maybe that's where the special sauce is.

I will have a look at the NORDIC code again, but from a pure methodological aspect, I think that optimal denoising (optimal-fro, -nuc, -op) are better.

I don't think I've seen the optimal thresholding methods used in fMRI/DWI papers before. Given the momentum NORDIC has been developing, I'd love to see a head-to-head comparison showing that these other approaches are better before the community goes all-in on NORDIC.

[paquiteau]

Well the only comparison I am aware for fMRI of is the one I made in our ISBI 2022 paper (where I used the NORDIC MATLAB version), and optimal-threshold outperformed the others competitors.

There is also https://www.sciencedirect.com/science/article/pii/S2666389924000539 for DWI. Here as well the Optimal threshold seems to perform better, but I am not an expert of DWI.

[tsalo]

Your fMRI paper is a great reference. I think replicating and extending the Moser paper or Logan Dowdle's 2023 paper would also be really useful.

The DWI paper you linked seems to be focused on uncombined (multi-channel) DWI data, but I think most people will care more about combined data.