Generating electrode group and information input with row keys and column keys not matching

[APaulk]

Hello,
I am trying to use pynwb to convert a large amount of Intan data into the NWB format. I cannot seem to figure out why I can't set up the device and add electrodes (see below). The code reads entries in a .csv database (df) that then enters in the values. I receive the below error and I don't know what it means?

I am also generating this electrode group (a 1024 channel grid) and then using that group in the below convert_to_nwb function to convert the data.

Any and all help would be hugely appreciated. Otherwise, we will have to enter this information for hundreds of files using this software (https://github.com/Intan-Technologies/IntanToNWB and excel sheet) which can lead to user errors.

Error:
ValueError: row data keys don't match available columns
you supplied 0 extra keys: set()
and were missing 1 keys: {'Serial Number- B16S1L2 Layout version- Rat150um_v7_6_1024ch_A'}

Thank you,
Angelique

Code with the full error far below. Sorry that I couldn't keep the same formatting as in jupyter notebook.
LengthFiles=len(AllRows)
RowNum=range(LengthFiles)

for x in RowNum:
print(x)
IdentifierInfo=df.iloc[x,15][2:len(df.iloc[x,15])]
ImpedanceFile=df.iloc[x,34]

Year=int(IdentifierInfo[len(IdentifierInfo)-8:len(IdentifierInfo)-4])
Day=int(IdentifierInfo[len(IdentifierInfo)-6:len(IdentifierInfo)-4])
Month=int(IdentifierInfo[len(IdentifierInfo)-8:len(IdentifierInfo)-6])
session_start_time = datetime(Year, Month, Day, 2, 30, 3, tzinfo=tz.gettz("US/Pacific"))

FileNameBasis=df.iloc[x,0][0:len(df.iloc[x,0])-4]

nwbfile = NWBFile(
    session_description=df.iloc[x,41] + ' ' + df.iloc[x,3] + ' ' + df.iloc[x,13] + ' ' + df.iloc[x,22] + ' ' + df.iloc[x,23] + ' ' + df.iloc[x,31],  # required
    identifier=FileNameBasis,  # required
    session_start_time=session_start_time,  # required
    session_id=df.iloc[x,0],  # optional
    experimenter="Youngbin Tchoe",  # optional
    lab="Dayeh lab, IEBL, http://iebl.ucsd.edu/",  # optional
    institution="University of California, San Diego",  # optional
    related_publications="DOI:10.1016/j.neuron.2016.12.011",  # optional
    )
print(nwbfile)

nwbfile.subject = Subject(
    subject_id=df.iloc[x,1],
    age=df.iloc[x,2],
    description=df.iloc[x,3] + ' ' + df.iloc[x,22],
    species=df.iloc[x,6],
    sex=df.iloc[x,5],
    weight=df.iloc[x,7],
    )

device = nwbfile.create_device(
        name=str(df.iloc[x,25]), 
        description=df.iloc[x,27] + ' ' + df.iloc[x,28] + ' ' + df.iloc[0,29], 
        manufacturer="IEBL, http://iebl.ucsd.edu/"
        )

csvfileImpedance=os.path.join(r'N:\\DeidentifiedNeuropixelsData\\UCSDBackup',df.iloc[x,35][2:len(df.iloc[x,35])], df.iloc[x,34])

file = open(csvfileImpedance, "r")
Impedance = list(csv.reader(file, delimiter=","))
file.close()

ChannelLabel=column(Impedance,0)[1:len(column(Impedance,0))]
ChannelLabel2=column(Impedance,1)[1:len(column(Impedance,1))]
Port=column(Impedance,2)[1:len(column(Impedance,2))]
Enabled=column(Impedance,3)[1:len(column(Impedance,3))]
ImpedanceValue=column(Impedance,4)[1:len(column(Impedance,4))]
ImpedancePhase=column(Impedance,5)[1:len(column(Impedance,5))]    

nwbfile.add_electrode_column(name=df.iloc[x,25], description=df.iloc[x,27] + df.iloc[x,28] + df.iloc[x,29])

nchannels = range(len(np.transpose(xcoords)))
electrode_counter = 0

electrode_group = nwbfile.create_electrode_group(
            name="PtNR grid",
            description="electrode contact in grid",
            device=device,
            location=df.iloc[x,31],
            )
filtering = 'default Intan'

for ishank in nchannels:        
        nwbfile.add_electrode(
            group=electrode_group,
            x=float(xcoordst[ishank]),  
            y=float(ycoordst[ishank]),
            z=float(0),
            imp=float(ImpedanceValue[ishank]),
            location=df.iloc[x,31],
            filtering=filtering,
            id=ishank,                
            rel_x=None, 
            rel_y=None, 
            rel_z=None, 
            reference=None, 
            enforce_unique_id=True,
        )
        electrode_counter += 1
        
FileNameNWBInfo=os.path.join(r'N:\\DeidentifiedNeuropixelsData\\UCSDBackup',df.iloc[x,35][2:len(df.iloc[x,35])],"IdentifierInfo.nwb")

io = NWBHDF5IO("IdentifierInfo.nwb", mode="w")
io.write(nwbfile)
io.close()
    
NWBFi=FileNameBasis +'.nwb'

convert_to_nwb(settings_filename=None,
                   intan_filename=os.path.join(r'N:\\DeidentifiedNeuropixelsData\\UCSDBackup',df.iloc[x,15][2:len(df.iloc[x,15])], df.iloc[x,0]),
                   nwb_filename=os.path.join(r'N:\\DeidentifiedNeuropixelsData\\UCSDBackup\\',IdentifierInfo,NWBFi),
                   session_description=df.iloc[x,41] + ' ' + df.iloc[x,3] + ' ' + df.iloc[x,13] + ' ' + df.iloc[x,22] + ' ' + df.iloc[x,23] + ' ' + df.iloc[x,31],
                   blocks_per_chunk=1000,
                   use_compression=True,
                   compression_level=4,
                   lowpass_description='default Intan filter settings: 0.02 Hz, The lower limit of the amplifier pass band has a simple one-pole high-pass filter characteristic',
                   highpass_description='default Intan filter settings: 20 kHz The upper limit of the amplifier pass band has a three-pole 3rd-order Butterworth low-pass filter',
                   merge_files=True,
                   subject=nwbfile.subject,
                   manual_start_time=None,
                  )

root pynwb.file.NWBFile at 0x1965047167464
Fields:
experimenter: ['Youngbin Tchoe']
file_create_date: [datetime.datetime(2022, 11, 9, 20, 33, 29, 576107, tzinfo=tzlocal())]
identifier: Baseline_saline_200317_095410
institution: University of California, San Diego
lab: Dayeh lab, IEBL, http://iebl.ucsd.edu/
related_publications: ['DOI:10.1016/j.neuron.2016.12.011']
session_description: Baseline recordings in saline Fine functional mapping of rat barrel cortex (right side) with micro-ECoG grids with 1024 channels, 150 µm pitch, 30 µm diameter Baseline recordings in saline Ketamine/Xylaxine (9:1) Acute Barrel cortex (right hemisphere)
session_id: Baseline_saline_200317_095410.rhd
session_start_time: 0317-03-17 02:30:03-07:53
timestamps_reference_time: 0317-03-17 02:30:03-07:53

---

ValueError Traceback (most recent call last)
in
81 rel_z=None,
82 reference=None,
---> 83 enforce_unique_id=True,
84 )
85 electrode_counter += 1

~\Anaconda3\lib\site-packages\hdmf\utils.py in func_call(*args, **kwargs)
643 def func_call(*args, **kwargs):
644 pargs = _check_args(args, kwargs)
--> 645 return func(args[0], **pargs)
646 else:
647 def func_call(*args, **kwargs):

~\Anaconda3\lib\site-packages\pynwb\file.py in add_electrode(self, **kwargs)
689 d.pop(col_name) # remove args from d if not set
690
--> 691 self.electrodes.add_row(**d)
692
693 @DocVal({'name': 'region', 'type': (slice, list, tuple), 'doc': 'the indices of the table'},

~\Anaconda3\lib\site-packages\hdmf\utils.py in func_call(*args, **kwargs)
643 def func_call(*args, **kwargs):
644 pargs = _check_args(args, kwargs)
--> 645 return func(args[0], **pargs)
646 else:
647 def func_call(*args, **kwargs):

~\Anaconda3\lib\site-packages\hdmf\common\table.py in add_row(self, **kwargs)
597 'row data keys don't match available columns',
598 'you supplied {} extra keys: {}'.format(len(extra_columns), extra_columns),
--> 599 'and were missing {} keys: {}'.format(len(missing_columns), missing_columns)
600 ])
601 )

ValueError: row data keys don't match available columns
you supplied 0 extra keys: set()
and were missing 1 keys: {'Serial Number- B16S1L2 Layout version- Rat150um_v7_6_1024ch_A'}

[rly]

What is the value of df.iloc[x,25]?

In this line:

nwbfile.add_electrode_column(name=df.iloc[x,25], description=df.iloc[x,27] + df.iloc[x,28] + df.iloc[x,29])

You add a new column to the electrodes table named whatever the value of df.iloc[x,25] is.

When you later add rows to the electrodes table, you must provide a value for this new custom column in nwbfile.add_electrode.

for ishank in nchannels:        
        nwbfile.add_electrode(
            group=electrode_group,
            x=float(xcoordst[ishank]),  
            y=float(ycoordst[ishank]),
            z=float(0),
            imp=float(ImpedanceValue[ishank]),
            location=df.iloc[x,31],
            filtering=filtering,
            id=ishank,                
            rel_x=None, 
            rel_y=None, 
            rel_z=None, 
            reference=None, 
            enforce_unique_id=True,
        )
        electrode_counter += 1

The error suggests that the new column name is "Serial Number- B16S1L2 Layout version- Rat150um_v7_6_1024ch_A"

[APaulk]

Hi, That explanation cleared things up. I just commented that 'nwbfile.add_electrode_column' line out. Thanks! Now just on to the adventure of figuring out what to enter into the "nwb_filename" in the 'convert_to_nwb' approach.
Thanks,
ACP