Catalog channel-to-stain mapping per source [discussions]

[bethac07]

#22 ; it's a reasonable request and not the last time we'll get it, for sure.

It's easy enough to do this from the files I already collected from all the partners for jump-cellpainting/datasets-private#20 , BUT we can't ensure consistency with that. Is there a non-annoying way to ensure consistency on this? I think for most partners the names should be stereotyped in the sense that Well A01 Site 1 is always the same name no matter what, so maybe reading the first row of the FileName columns in each load_data.csv to see if it's consistent or not?

[bethac07]

I added the non-consistency-checked version to the same spreadsheet

[bethac07]

My proposed approach will definitely not work for source_8 unparsed, because the microscope does that "random string" append which I HATE SO MUCH; but one could use a regex to search up through w# .

[bethac07]

I probably need jump-cellpainting/datasets-private#31 to resolve the TBDs - the Phenix XML directly gives the order of brightfield plane acquisition but the CV8000 mes does not (it gives total span and step size but not order). I've reached out in Teams but if no one there knows (or if it's configurable by company) I'll need to check the channel order they gave to CellProfiler and hope they did it correctly.

[shntnu]

I believe this was addressed in #21 and #25

[bethac07]

Nope, this was not, this is related but separate. (That was "what are the filter parameters for each stain and how many images are there" this is "what channel number does each stain correspond to")

Charles pointed me at the mlf files, which got me the info I needed for the CV8000 partners, so all the TBDs are filled in for the table now, with the caveat of "this is for one plate each per partner" - I have no idea if this is consistent (I have no especial reason to believe that it isn't, but we know on our scope this has changed sometimes batch to batch, and this project has not been the poster-project for consistency).

I can probably write some code to do consistency checking (there are only 3 kinds of files to check) but it's not very top of my list at the moment. I think I'd rather do that than to use the Load Data CSVs (other than for the source 8, where we have no choice, but that's one where I'm pretty confident things will have been done correctly, and since there is no brightfield, it should be visually quite obvious what all the channels are), just because in some cases we know that there may have been a wide time and/or person gap between the imaging and the person setting up CellProfiler, and so it's hard to rule out mistakes at that stage. If we do find for any sources that the ordering was NOT consistent across batches, we can then verify their load_data CSVs accurate reflect that.

Ordinarily, I wouldn't be so insane-nitpicky, but this data set is going to be used for a lot of stuff (by ourselves and others) going forward, we want to make dang sure it's thoroughly characterized.

[ErinWeisbart]

Since I was wrangling .csv's I thought I'd take a quick check at consistency between the load_data.csv's. Good news is that everything other than source_1 is consistent between .csv's. I checked that the value in Well B4 Site 2 that included the channel identifier matched between the original .csv's and the .csv's that I fixed (as a sanity check) and matched from the first .csv's in the source to all other .csv's (to answer this open question, from the .csv perspective).

Different sources required slightly different parsing so here's the working code for all passing sources.

source_2

import pandas as pd
import os
root = '/Users/eweisbar/Desktop/fixed'
origroot = '/Users/eweisbar/Desktop/fixloaddata_orig'

source = 'source_2' # 3 BF

first = True
for batch in [x for x in os.listdir(os.path.join(root,source, 'workspace','load_data_csv')) if '.DS' not in x]:
    for plate in [x for x in os.listdir(os.path.join(root, source,'workspace','load_data_csv',batch)) if '.DS' not in x]:
        fixcsv = pd.read_csv(os.path.join(root, source,'workspace','load_data_csv',batch,plate,'load_data.csv.gz'))
        #fixillumcsv = pd.read_csv(os.path.join(root, source,'workspace','load_data_csv',batch,plate,'load_data_with_illum.csv.gz'))
        origcsv = pd.read_csv(os.path.join(origroot, source,'workspace','load_data_csv',batch,plate,'load_data.csv.gz'))
        origcsv.columns = [x.replace('Image_','') for x in origcsv.columns]
        #origillumcsv = pd.read_csv(os.path.join(origroot, source,'workspace','load_data_csv',batch,plate,'load_data_with_illum.csv'))
        if first:
            check_dict = {'AGP':fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigAGP'].values[0].split('_',1)[1],
                          'Mito':fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigMito'].values[0].split('_',1)[1],
                          'DNA':fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigDNA'].values[0].split('_',1)[1],
                          'ER':fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigER'].values[0].split('_',1)[1],
                          'RNA':fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigRNA'].values[0].split('_',1)[1],
                          'Brightfield':fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigBrightfield'].values[0].split('_',1)[1],
                          'Brightfield_H':fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigBrightfield_H'].values[0].split('_',1)[1],
                          'Brightfield_L':fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigBrightfield_L'].values[0].split('_',1)[1],
                          }
        try:
            assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigAGP'].values[0].split('_',1)[1] == origcsv.loc[origcsv['Metadata_Well']=='B04'].loc[origcsv['Metadata_Field']==2,'FileName_OrigAGP'].values[0].split('_',1)[1]
            assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigMito'].values[0].split('_',1)[1] == origcsv.loc[origcsv['Metadata_Well']=='B04'].loc[origcsv['Metadata_Field']==2,'FileName_OrigMito'].values[0].split('_',1)[1]
            assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigDNA'].values[0].split('_',1)[1] == origcsv.loc[origcsv['Metadata_Well']=='B04'].loc[origcsv['Metadata_Field']==2,'FileName_OrigDNA'].values[0].split('_',1)[1]
            assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigER'].values[0].split('_',1)[1] == origcsv.loc[origcsv['Metadata_Well']=='B04'].loc[origcsv['Metadata_Field']==2,'FileName_OrigER'].values[0].split('_',1)[1]
            assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigRNA'].values[0].split('_',1)[1] == origcsv.loc[origcsv['Metadata_Well']=='B04'].loc[origcsv['Metadata_Field']==2,'FileName_OrigRNA'].values[0].split('_',1)[1]
            assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigBrightfield'].values[0].split('_',1)[1] == origcsv.loc[origcsv['Metadata_Well']=='B04'].loc[origcsv['Metadata_Field']==2,'FileName_OrigBrightfield'].values[0].split('_',1)[1]
            assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigBrightfield_H'].values[0].split('_',1)[1] == origcsv.loc[origcsv['Metadata_Well']=='B04'].loc[origcsv['Metadata_Field']==2,'FileName_OrigBrightfield_H'].values[0].split('_',1)[1]
            assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigBrightfield_L'].values[0].split('_',1)[1] == origcsv.loc[origcsv['Metadata_Well']=='B04'].loc[origcsv['Metadata_Field']==2,'FileName_OrigBrightfield_L'].values[0].split('_',1)[1]
        except:
            print(batch,plate, 'failed orig vs fix check')  
            print(fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigAGP'].values[0].split('_',1)[1])
            print(origcsv.loc[origcsv['Metadata_Well']=='B04'].loc[origcsv['Metadata_Field']==2,'FileName_OrigAGP'].values[0].split('_',1)[1])
        try:
            assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigAGP'].values[0].split('_',1)[1] == check_dict['AGP']
            assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigMito'].values[0].split('_',1)[1] == check_dict['Mito']
            assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigDNA'].values[0].split('_',1)[1] == check_dict['DNA']
            assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigER'].values[0].split('_',1)[1] == check_dict['ER']
            assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigRNA'].values[0].split('_',1)[1] == check_dict['RNA']
            assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigBrightfield'].values[0].split('_',1)[1] == check_dict['Brightfield']
            assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigBrightfield_H'].values[0].split('_',1)[1] == check_dict['Brightfield_H']
            assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigBrightfield_L'].values[0].split('_',1)[1] == check_dict['Brightfield_L']        
        except:
            print (batch,plate, 'failed consistency check')
            print (fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigAGP'].values[0].split('_',1)[1]) 
            print(check_dict['AGP'])
        first = False

source_3, source_4, source_9

import pandas as pd
import os
root = '/Users/eweisbar/Desktop/fixed'
origroot = '/Users/eweisbar/Desktop/fixloaddata_orig'

source = 'source_3' # some 0 BF, some 3 BF
#source = 'source_4' # 3 BF
#source = 'source_9' # no BF

first = True
for batch in [x for x in os.listdir(os.path.join(root,source, 'workspace','load_data_csv')) if '.DS' not in x]:
    for plate in [x for x in os.listdir(os.path.join(root, source,'workspace','load_data_csv',batch)) if '.DS' not in x]:
        fixcsv = pd.read_csv(os.path.join(root, source,'workspace','load_data_csv',batch,plate,'load_data.csv.gz'))
        #fixillumcsv = pd.read_csv(os.path.join(root, source,'workspace','load_data_csv',batch,plate,'load_data_with_illum.csv.gz'))
        origcsv = pd.read_csv(os.path.join(origroot, source,'workspace','load_data_csv',batch,plate,'load_data.csv.gz'))
        if [x for x in origcsv.columns if 'Bright' in x] != []:
            has_BF = True
        else:
            has_BF = False
        #origillumcsv = pd.read_csv(os.path.join(origroot, source,'workspace','load_data_csv',batch,plate,'load_data_with_illum.csv.gz'))
        if first:
            check_dict = {'AGP':fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigAGP'].values[0],
                          'Mito':fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigMito'].values[0],
                          'DNA':fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigDNA'].values[0],
                          'ER':fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigER'].values[0],
                          'RNA':fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigRNA'].values[0]}
            if has_BF:
                check_dict = check_dict | {'Brightfield':fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigBrightfield'].values[0],
                          'Brightfield_H':fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigBrightfield_H'].values[0],
                          'Brightfield_L':fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigBrightfield_L'].values[0],}

        try:
            assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigAGP'].values[0] == origcsv.loc[origcsv['Metadata_Well']=='B04'].loc[origcsv['Metadata_Site']==2,'FileName_OrigAGP'].values[0]
            assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigMito'].values[0] == origcsv.loc[origcsv['Metadata_Well']=='B04'].loc[origcsv['Metadata_Site']==2,'FileName_OrigMito'].values[0]
            assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigDNA'].values[0] == origcsv.loc[origcsv['Metadata_Well']=='B04'].loc[origcsv['Metadata_Site']==2,'FileName_OrigDNA'].values[0]
            assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigER'].values[0] == origcsv.loc[origcsv['Metadata_Well']=='B04'].loc[origcsv['Metadata_Site']==2,'FileName_OrigER'].values[0]
            assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigRNA'].values[0] == origcsv.loc[origcsv['Metadata_Well']=='B04'].loc[origcsv['Metadata_Site']==2,'FileName_OrigRNA'].values[0]
            if has_BF:
                assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigBrightfield'].values[0] == origcsv.loc[origcsv['Metadata_Well']=='B04'].loc[origcsv['Metadata_Site']==2,'FileName_OrigBrightfield'].values[0]
                assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigBrightfield_H'].values[0] == origcsv.loc[origcsv['Metadata_Well']=='B04'].loc[origcsv['Metadata_Site']==2,'FileName_OrigBrightfield_H'].values[0]
                assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigBrightfield_L'].values[0] == origcsv.loc[origcsv['Metadata_Well']=='B04'].loc[origcsv['Metadata_Site']==2,'FileName_OrigBrightfield_L'].values[0]
        except:
            print(batch,plate, 'failed orig vs fix check')  
            print(fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigAGP'].values[0])
            print(origcsv.loc[origcsv['Metadata_Well']=='B04'].loc[origcsv['Metadata_Site']==2,'FileName_OrigAGP'].values[0])
        try:
            assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigAGP'].values[0] == check_dict['AGP']
            assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigMito'].values[0] == check_dict['Mito']
            assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigDNA'].values[0] == check_dict['DNA']
            assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigER'].values[0] == check_dict['ER']
            assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigRNA'].values[0] == check_dict['RNA']
            if has_BF:
                assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigBrightfield'].values[0] == check_dict['Brightfield']
                assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigBrightfield_H'].values[0] == check_dict['Brightfield_H']
                assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigBrightfield_L'].values[0] == check_dict['Brightfield_L']        

        except:
            print (batch,plate, 'failed consistency check')
            print (fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigAGP'].values[0]) 
            print(check_dict['AGP'])
        first = False

source_5

source = 'source_5' # 3 BF
first = True
for batch in [x for x in os.listdir(os.path.join(root,source, 'workspace','load_data_csv')) if '.DS' not in x]:
    for plate in [x for x in os.listdir(os.path.join(root, source,'workspace','load_data_csv',batch)) if '.DS' not in x]:
        fixcsv = pd.read_csv(os.path.join(root, source,'workspace','load_data_csv',batch,plate,'load_data.csv.gz'))
        fixillumcsv = pd.read_csv(os.path.join(root, source,'workspace','load_data_csv',batch,plate,'load_data_with_illum.csv.gz'))
        origcsv = pd.read_csv(os.path.join(origroot, source,'workspace','load_data_csv',batch,plate,'load_data.csv.gz'))
        origillumcsv = pd.read_csv(os.path.join(origroot, source,'workspace','load_data_csv',batch,plate,'load_data_with_illum.csv.gz'))
        if first:
            check_dict = {'AGP':fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigAGP'].values[0].split('_',2)[2],
                          'Mito':fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigMito'].values[0].split('_',2)[2],
                          'DNA':fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigDNA'].values[0].split('_',2)[2],
                          'ER':fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigER'].values[0].split('_',2)[2],
                          'RNA':fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigRNA'].values[0].split('_',2)[2],
                          'Brightfield':fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigBrightfield'].values[0].split('_',2)[2],
                          'Brightfield_H':fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigBrightfield_H'].values[0].split('_',2)[2],
                          'Brightfield_L':fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigBrightfield_L'].values[0].split('_',2)[2]
                          }
        try:
            assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigAGP'].values[0].split('_',2)[2] == origcsv.loc[origcsv['Metadata_Well']=='B04'].loc[origcsv['Metadata_Site']==2,'FileName_OrigAGP'].values[0].split('_',2)[2]
            assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigMito'].values[0].split('_',2)[2] == origcsv.loc[origcsv['Metadata_Well']=='B04'].loc[origcsv['Metadata_Site']==2,'FileName_OrigMito'].values[0].split('_',2)[2]
            assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigDNA'].values[0].split('_',2)[2] == origcsv.loc[origcsv['Metadata_Well']=='B04'].loc[origcsv['Metadata_Site']==2,'FileName_OrigDNA'].values[0].split('_',2)[2]
            assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigER'].values[0].split('_',2)[2] == origcsv.loc[origcsv['Metadata_Well']=='B04'].loc[origcsv['Metadata_Site']==2,'FileName_OrigER'].values[0].split('_',2)[2]
            assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigRNA'].values[0].split('_',2)[2] == origcsv.loc[origcsv['Metadata_Well']=='B04'].loc[origcsv['Metadata_Site']==2,'FileName_OrigRNA'].values[0].split('_',2)[2]
            assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigBrightfield'].values[0].split('_',2)[2] == origcsv.loc[origcsv['Metadata_Well']=='B04'].loc[origcsv['Metadata_Site']==2,'FileName_OrigBrightfield'].values[0].split('_',2)[2]
            assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigBrightfield_H'].values[0].split('_',2)[2] == origcsv.loc[origcsv['Metadata_Well']=='B04'].loc[origcsv['Metadata_Site']==2,'FileName_OrigBrightfield_H'].values[0].split('_',2)[2]
            assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigBrightfield_L'].values[0].split('_',2)[2] == origcsv.loc[origcsv['Metadata_Well']=='B04'].loc[origcsv['Metadata_Site']==2,'FileName_OrigBrightfield_L'].values[0].split('_',2)[2]
        except:
            print(batch,plate, 'failed orig vs fix check')  
            print(fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigAGP'].values[0].split('_',2)[2])
            print(origcsv.loc[origcsv['Metadata_Well']=='B04'].loc[origcsv['Metadata_Site']==2,'FileName_OrigAGP'].values[0].split('_',2)[2])
        try:
            assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigAGP'].values[0].split('_',2)[2] == check_dict['AGP']
            assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigMito'].values[0].split('_',2)[2] == check_dict['Mito']
            assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigDNA'].values[0].split('_',2)[2] == check_dict['DNA']
            assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigER'].values[0].split('_',2)[2] == check_dict['ER']
            assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigRNA'].values[0].split('_',2)[2] == check_dict['RNA']
            assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigBrightfield'].values[0].split('_',2)[2] == check_dict['Brightfield']
            assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigBrightfield_H'].values[0].split('_',2)[2] == check_dict['Brightfield_H']
            assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigBrightfield_L'].values[0].split('_',2)[2] == check_dict['Brightfield_L']
        except:
            print (batch,plate, 'failed consistency check')
            print (fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigAGP'].values[0]) 
            print(check_dict['AGP'])
        first = False

source_6, source_7, source_10, source_13

import pandas as pd
import os
root = '/Users/eweisbar/Desktop/fixed'
origroot = '/Users/eweisbar/Desktop/fixloaddata_orig'

#source = 'source_6' # 3 BF
#source = 'source_7' # no BF
#source = 'source_10' # 3 BF
source = 'source_13' # no BF

first = True
for batch in [x for x in os.listdir(os.path.join(root,source, 'workspace','load_data_csv')) if '.DS' not in x]:
    for plate in [x for x in os.listdir(os.path.join(root, source,'workspace','load_data_csv',batch)) if '.DS' not in x]:
        fixcsv = pd.read_csv(os.path.join(root, source,'workspace','load_data_csv',batch,plate,'load_data.csv.gz'))
        #fixillumcsv = pd.read_csv(os.path.join(root, source,'workspace','load_data_csv',batch,plate,'load_data_with_illum.csv.gz'))
        origcsv = pd.read_csv(os.path.join(origroot, source,'workspace','load_data_csv',batch,plate,'load_data.csv.gz'))
        origcsv.columns = [x.replace('Image_','') for x in origcsv.columns]
        if [x for x in origcsv.columns if 'Bright' in x] != []:
            has_BF = True
        else:
            has_BF = False
        #origillumcsv = pd.read_csv(os.path.join(origroot, source,'workspace','load_data_csv',batch,plate,'load_data_with_illum.csv'))
        if first:
            check_dict = {'AGP':fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigAGP'].values[0].split('_',1)[1],
                          'Mito':fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigMito'].values[0].split('_',1)[1],
                          'DNA':fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigDNA'].values[0].split('_',1)[1],
                          'ER':fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigER'].values[0].split('_',1)[1],
                          'RNA':fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigRNA'].values[0].split('_',1)[1],}
            if has_BF:
                check_dict = check_dict | {'Brightfield':fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigBrightfield'].values[0].split('_',1)[1],
                          'Brightfield_H':fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigBrightfield_H'].values[0].split('_',1)[1],
                          'Brightfield_L':fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigBrightfield_L'].values[0].split('_',1)[1],}

        try:
            assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigAGP'].values[0].split('_',1)[1] == origcsv.loc[origcsv['Metadata_Well']=='B04'].loc[origcsv['Metadata_Site']==2,'FileName_OrigAGP'].values[0].split('_',1)[1]
            assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigMito'].values[0].split('_',1)[1] == origcsv.loc[origcsv['Metadata_Well']=='B04'].loc[origcsv['Metadata_Site']==2,'FileName_OrigMito'].values[0].split('_',1)[1]
            assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigDNA'].values[0].split('_',1)[1] == origcsv.loc[origcsv['Metadata_Well']=='B04'].loc[origcsv['Metadata_Site']==2,'FileName_OrigDNA'].values[0].split('_',1)[1]
            assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigER'].values[0].split('_',1)[1] == origcsv.loc[origcsv['Metadata_Well']=='B04'].loc[origcsv['Metadata_Site']==2,'FileName_OrigER'].values[0].split('_',1)[1]
            assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigRNA'].values[0].split('_',1)[1] == origcsv.loc[origcsv['Metadata_Well']=='B04'].loc[origcsv['Metadata_Site']==2,'FileName_OrigRNA'].values[0].split('_',1)[1]
            if has_BF:
                assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigBrightfield'].values[0].split('_',1)[1] == origcsv.loc[origcsv['Metadata_Well']=='B04'].loc[origcsv['Metadata_Site']==2,'FileName_OrigBrightfield'].values[0].split('_',1)[1]
                assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigBrightfield_H'].values[0].split('_',1)[1] == origcsv.loc[origcsv['Metadata_Well']=='B04'].loc[origcsv['Metadata_Site']==2,'FileName_OrigBrightfield_H'].values[0].split('_',1)[1]
                assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigBrightfield_L'].values[0].split('_',1)[1] == origcsv.loc[origcsv['Metadata_Well']=='B04'].loc[origcsv['Metadata_Site']==2,'FileName_OrigBrightfield_L'].values[0].split('_',1)[1]
        except:
            print(batch,plate, 'failed orig vs fix check')  
            print(fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigAGP'].values[0].split('_',1)[1])
            print(origcsv.loc[origcsv['Metadata_Well']=='B04'].loc[origcsv['Metadata_Site']==2,'FileName_OrigAGP'].values[0].split('_',1)[1])
        
        try:
            assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigAGP'].values[0].split('_',1)[1] == check_dict['AGP']
            assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigMito'].values[0].split('_',1)[1] == check_dict['Mito']
            assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigDNA'].values[0].split('_',1)[1] == check_dict['DNA']
            assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigER'].values[0].split('_',1)[1] == check_dict['ER']
            assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigRNA'].values[0].split('_',1)[1] == check_dict['RNA']
            if has_BF:
                assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigBrightfield'].values[0].split('_',1)[1] == check_dict['Brightfield']
                assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigBrightfield_H'].values[0].split('_',1)[1] == check_dict['Brightfield_H']
                assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigBrightfield_L'].values[0].split('_',1)[1] == check_dict['Brightfield_L']        

        except:
            print (batch,plate, 'failed consistency check')
            print (fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigAGP'].values[0].split('_',1)[1]) 
            print(check_dict['AGP'])
        first = False

source_8

import pandas as pd
import os
root = '/Users/eweisbar/Desktop/fixed'
origroot = '/Users/eweisbar/Desktop/fixloaddata_orig'

source = 'source_8' # no BF
first = True
for batch in [x for x in os.listdir(os.path.join(root,source, 'workspace','load_data_csv')) if '.DS' not in x]:
    for plate in [x for x in os.listdir(os.path.join(root, source,'workspace','load_data_csv',batch)) if '.DS' not in x]:
        fixcsv = pd.read_csv(os.path.join(root, source,'workspace','load_data_csv',batch,plate,'load_data.csv.gz'))
        #fixillumcsv = pd.read_csv(os.path.join(root, source,'workspace','load_data_csv',batch,plate,'load_data_with_illum.csv.gz'))
        origcsv = pd.read_csv(os.path.join(origroot, source,'workspace','load_data_csv',batch,plate,'load_data.csv.gz'))
        origcsv.columns = [x.replace('Image_','') for x in origcsv.columns]
        #origillumcsv = pd.read_csv(os.path.join(origroot, source,'workspace','load_data_csv',batch,plate,'load_data_with_illum.csv'))
        if first:
            check_dict = {'AGP':fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigAGP'].values[0].split('_w')[1][0],
                          'Mito':fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigMito'].values[0].split('_w')[1][0],
                          'DNA':fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigDNA'].values[0].split('_w')[1][0],
                          'ER':fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigER'].values[0].split('_w')[1][0],
                          'RNA':fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigRNA'].values[0].split('_w')[1][0]}
        try:
            assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigAGP'].values[0].split('_w')[1][0] == origcsv.loc[origcsv['Metadata_Well']=='B04'].loc[origcsv['Metadata_Site']==2,'FileName_OrigAGP'].values[0].split('_w')[1][0]
            assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigMito'].values[0].split('_w')[1][0] == origcsv.loc[origcsv['Metadata_Well']=='B04'].loc[origcsv['Metadata_Site']==2,'FileName_OrigMito'].values[0].split('_w')[1][0]
            assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigDNA'].values[0].split('_w')[1][0] == origcsv.loc[origcsv['Metadata_Well']=='B04'].loc[origcsv['Metadata_Site']==2,'FileName_OrigDNA'].values[0].split('_w')[1][0]
            assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigER'].values[0].split('_w')[1][0] == origcsv.loc[origcsv['Metadata_Well']=='B04'].loc[origcsv['Metadata_Site']==2,'FileName_OrigER'].values[0].split('_w')[1][0]
            assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigRNA'].values[0].split('_w')[1][0] == origcsv.loc[origcsv['Metadata_Well']=='B04'].loc[origcsv['Metadata_Site']==2,'FileName_OrigRNA'].values[0].split('_w')[1][0]
        except:
            print(batch,plate, 'failed orig vs fix check')  
            print(fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigAGP'].values[0].split('_w')[1][0])
            print(origcsv.loc[origcsv['Metadata_Well']=='B04'].loc[origcsv['Metadata_Site']==2,'FileName_OrigAGP'].values[0].split('_w')[1][0])
        try:
            assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigAGP'].values[0].split('_w')[1][0] == check_dict['AGP']
            assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigMito'].values[0].split('_w')[1][0] == check_dict['Mito']
            assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigDNA'].values[0].split('_w')[1][0] == check_dict['DNA']
            assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigER'].values[0].split('_w')[1][0] == check_dict['ER']
            assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigRNA'].values[0].split('_w')[1][0] == check_dict['RNA']
        except:
            print (batch,plate, 'failed consistency check')
            print (fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigAGP'].values[0].split('_w')[1][0]) 
            print(check_dict['AGP'])
        first = False

source_11

import pandas as pd
import os
root = '/Users/eweisbar/Desktop/fixed'
origroot = '/Users/eweisbar/Desktop/fixloaddata_orig'

source = 'source_11' # 1 BF in some batches, 0 BF in others
first = True
for batch in [x for x in os.listdir(os.path.join(root,source, 'workspace','load_data_csv')) if '.DS' not in x]:
    for plate in [x for x in os.listdir(os.path.join(root, source,'workspace','load_data_csv',batch)) if '.DS' not in x]:
        fixcsv = pd.read_csv(os.path.join(root, source,'workspace','load_data_csv',batch,plate,'load_data.csv.gz'))
        #fixillumcsv = pd.read_csv(os.path.join(root, source,'workspace','load_data_csv',batch,plate,'load_data_with_illum.csv.gz'))
        if plate not in ['EC000038','EC000066','EC000070','EC000156']:
            origcsv = pd.read_csv(os.path.join(origroot, source,'workspace','load_data_csv',batch,plate,'load_data.csv.gz'))
            if [x for x in origcsv.columns if 'Bright' in x] != []:
                has_BF = True
            else:
                has_BF = False
            #origillumcsv = pd.read_csv(os.path.join(origroot, source,'workspace','load_data_csv',batch,plate,'load_data_with_illum.csv.gz'))
        else: # prefer to check in orig, not present for these plates
            if [x for x in fixcsv.columns if 'Bright' in x] != []:
                has_BF = True
            else:
                has_BF = False            
        if first:
            check_dict = {'AGP':fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigAGP'].values[0],
                          'Mito':fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigMito'].values[0],
                          'DNA':fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigDNA'].values[0],
                          'ER':fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigER'].values[0],
                          'RNA':fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigRNA'].values[0],}
            if has_BF:
                check_dict = check_dict | {'Brightfield':fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigBrightfield'].values[0]}

        if plate not in ['EC000038','EC000066','EC000070','EC000156']:    
            try:
                assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigAGP'].values[0] == origcsv.loc[origcsv['Metadata_Well']=='B04'].loc[origcsv['Metadata_Site']==2,'FileName_OrigAGP'].values[0]
                assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigMito'].values[0] == origcsv.loc[origcsv['Metadata_Well']=='B04'].loc[origcsv['Metadata_Site']==2,'FileName_OrigMito'].values[0]
                assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigDNA'].values[0] == origcsv.loc[origcsv['Metadata_Well']=='B04'].loc[origcsv['Metadata_Site']==2,'FileName_OrigDNA'].values[0]
                assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigER'].values[0] == origcsv.loc[origcsv['Metadata_Well']=='B04'].loc[origcsv['Metadata_Site']==2,'FileName_OrigER'].values[0]
                assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigRNA'].values[0] == origcsv.loc[origcsv['Metadata_Well']=='B04'].loc[origcsv['Metadata_Site']==2,'FileName_OrigRNA'].values[0]
                if has_BF:
                    assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigBrightfield'].values[0] == origcsv.loc[origcsv['Metadata_Well']=='B04'].loc[origcsv['Metadata_Site']==2,'FileName_OrigBrightfield'].values[0]
            except:
                print(batch,plate, 'failed orig vs fix check')  
                print(fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigAGP'].values[0])
                print(origcsv.loc[origcsv['Metadata_Well']=='B04'].loc[origcsv['Metadata_Site']==2,'FileName_OrigAGP'].values[0])
        try:
            assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigAGP'].values[0] == check_dict['AGP']
            assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigMito'].values[0] == check_dict['Mito']
            assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigDNA'].values[0] == check_dict['DNA']
            assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigER'].values[0] == check_dict['ER']
            assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigRNA'].values[0] == check_dict['RNA']
            if has_BF:
                assert fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigBrightfield'].values[0] == check_dict['Brightfield']
        except:
            print (batch,plate, 'failed consistency check')
            print (fixcsv.loc[fixcsv['Metadata_Well']=='B04'].loc[fixcsv['Metadata_Site']==2,'FileName_OrigAGP'].values[0]) 
            print(check_dict['AGP'])
        first = False

I didn't deep dive into source_1, but will note there are likely multiple configurations as at a glance AGP is parsing as ch3, ch4, or ch5 depending on the plate.

[shntnu]

I'm moving this to the public repo because there is no identifiable information here