Consistency of double underscore in BackgroundOnly names

[bethac07]

It looks like in our master pipeline, in 3 of the 8 channels (AGP, BFHigh, and ER), there was a typo in the MaskImage step such that the names of these channels have two underscores between the channel name and the words BackgroundOnly. How this looks for one measurement is below - there are 11 image intensity measurements, so overall, this affects 33 columns in total (22 for partners without a BFHigh channel).

In theory, everyone built their pipeline off of ours, but we had no way to enforce that, obviously, so we can't be sure without checking who does and who does not have the typo - maybe everyone does, but maybe not. Questions then

• Do we think this is something that's going to break things? I assume we're not "fuzzy" matching the column names
• Is this something worth fixing if it isn't consistent across all partners?
• Is this something that is worth fixing even if it is consistent across all partners? (I assume no, this late in the game?)

I ask because I'm in the process of writing the column fix code for source_15 (where we plan to rewrite all column names containing RNA to say ER instead), and I need to know if I should fix the RNA_Background measurements to ER_Background or ER__Background

Image_Intensity_LowerQuartileIntensity_AGP__BackgroundOnly 
Image_Intensity_LowerQuartileIntensity_BFHigh__BackgroundOnly
Image_Intensity_LowerQuartileIntensity_BFLow_BackgroundOnly
Image_Intensity_LowerQuartileIntensity_Brightfield_BackgroundOnly
Image_Intensity_LowerQuartileIntensity_DNA_BackgroundOnly 
Image_Intensity_LowerQuartileIntensity_ER__BackgroundOnly 
Image_Intensity_LowerQuartileIntensity_Mito_BackgroundOnly 
Image_Intensity_LowerQuartileIntensity_RNA_BackgroundOnly

[johnarevalo]

When matching columns in the aggregated profiles (details in jump-cellpainting/datasets-private#25) we created the "mandatory columns list" (it has 22 names with __). We check all of these columns exist in the aggregated profiles before we push them to cellpainting-gallery. I think it'd be good to make source_15 consistent with such list.

If we decide to change it, then we should update the already released profiles as well.

[bethac07]

Ok! If our previous validation checks confirm that the error is at least consistent, then I will plan when I get the go ahead to run the source_15 fix code, to convert RNA_Background to ER__Background . I can live with messy-but-consistent, for sure, but we may want to think about if it's worth fixing on the whole set once we have a better versioning system in place - someone who wants to programmatically construct and find column names, for instance, will not be able to do so in the dataset for these 22/33 columns.

[shntnu]

@bethac07 this is far from urgent or important, but if source_15 is in recent memory, then it might be the right time to close this out (or add a note on what's next for this issue)