How to use default values for DynamicTable when creating extension?

[tuanpham96]

I am trying to define a DynamicTable in an extension, and want to have certain required columns have default values so that when using add_row, those columns would take such values if not defined. I believe using a custom API would achieve this but I was wondering if that could already be done at the extension creation step. Here's what I've tried.

Minimal example

Step 1. Create extension

from pynwb.spec import NWBNamespaceBuilder, NWBGroupSpec, NWBDatasetSpec
name = 'ndx-testdyntbl'
ns_path = name + ".namespace.yaml"
ext_source = name + ".extensions.yaml"

ns_builder = NWBNamespaceBuilder(name + ' extensions', name, version='0.1.0')
ns_builder.include_type('DynamicTable', namespace='core')
ns_builder.include_type('VectorData', namespace='core')

my_table = NWBGroupSpec(
    default_name='my_table',
    neurodata_type_def='TestDynTbl',
    neurodata_type_inc='DynamicTable',
    doc='Test DynamicTable with defaults',
    datasets=[
        NWBDatasetSpec(
            name='col_1', doc='Column 1 [no default, required]', dtype='text',
            neurodata_type_inc='VectorData',
        ),
        NWBDatasetSpec(
            name='col_2', doc='Column 2 [has default, required]', dtype='float',
            neurodata_type_inc='VectorData',
            default_value=3.14,
        ),
        NWBDatasetSpec(
            name='col_3', doc='Column 3 [has default, not required]', dtype='int',
            neurodata_type_inc='VectorData',
            default_value=0,
            quantity='?'
        ),
        NWBDatasetSpec(
            name='col_4', doc='Column 4 [not default, not required]', dtype='int',
            neurodata_type_inc='VectorData',
            quantity='?'
        ),
    ]
)

ns_builder.add_spec(ext_source, my_table)
ns_builder.export(ns_path)

Spec files

# ndx-testdyntbl.extensions.yaml
groups:
- neurodata_type_def: TestDynTbl
  neurodata_type_inc: DynamicTable
  default_name: my_table
  doc: Test DynamicTable with defaults
  datasets:
  - name: col_1
    neurodata_type_inc: VectorData
    dtype: text
    doc: Column 1 [no default, required]
  - name: col_2
    neurodata_type_inc: VectorData
    dtype: float
    default_value: 3.14
    doc: Column 2 [has default, required]
  - name: col_3
    neurodata_type_inc: VectorData
    dtype: int
    default_value: 0
    doc: Column 3 [has default, not required]
    quantity: '?'
  - name: col_4
    neurodata_type_inc: VectorData
    dtype: int
    doc: Column 4 [not default, not required]
    quantity: '?'

# ndx-testdyntbl.namespace.yaml
namespaces:
- doc: ndx-testdyntbl extensions
  name: ndx-testdyntbl
  schema:
  - namespace: core
    neurodata_types:
    - DynamicTable
    - VectorData
  - source: ndx-testdyntbl.extensions.yaml
  version: 0.1.0

Step 2. Load extension

from datetime import datetime
from dateutil import tz

from pynwb import NWBFile, load_namespaces, get_class,NWBHDF5IO
from hdmf.utils import get_docval

load_namespaces('ndx-testdyntbl.namespace.yaml')
TestDynTbl = get_class('TestDynTbl', 'ndx-testdyntbl')

Output of get_docval(TestDynTbl.__init__)

({'name': 'description',
  'type': str,
  'doc': 'a description of what is in this table'},
 {'name': 'name',
  'type': str,
  'doc': 'the name of this table',
  'default': 'my_table'},
 {'name': 'id',
  'type': (numpy.ndarray,
   list,
   tuple,
   h5py._hl.dataset.Dataset,
   hdmf.utils.StrDataset,
   hdmf.query.HDMFDataset,
   hdmf.data_utils.AbstractDataChunkIterator,
   hdmf.data_utils.DataIO,
   hdmf.common.table.ElementIdentifiers),
  'doc': 'the identifiers for this table',
  'default': None},
 {'name': 'columns',
  'type': (tuple, list),
  'doc': 'the columns in this table',
  'default': None},
 {'name': 'colnames',
  'type': (numpy.ndarray,
   list,
   tuple,
   h5py._hl.dataset.Dataset,
   hdmf.utils.StrDataset,
   hdmf.query.HDMFDataset,
   hdmf.data_utils.AbstractDataChunkIterator),
  'doc': 'the ordered names of the columns in this table. columns must also be provided.',
  'default': None},
 {'name': 'target_tables',
  'doc': 'dict mapping DynamicTableRegion column name to the table that the DTR points to. The column is added to the table if it is not already present (i.e., when it is optional).',
  'type': dict,
  'default': None})

However, printing out TestDynTbl.__columns__ reveals no default* fields for these column metadata:

({'name': 'col_1',
  'description': 'Column 1 [no default, required]',
  'required': True,
  'class': hdmf.common.table.VectorData},
 {'name': 'col_2',                                          # <--- missing  `default[_value]=3.14`
  'description': 'Column 2 [has default, required]', 
  'required': True,
  'class': hdmf.common.table.VectorData},
 {'name': 'col_3',
  'description': 'Column 3 [has default, not required]',    # <--- missing  `default[_value]=0` 
  'required': False,
  'class': hdmf.common.table.VectorData},
 {'name': 'col_4',
  'description': 'Column 4 [not default, not required]',
  'required': False,
  'class': hdmf.common.table.VectorData})

Step 3. Usage & issues

Attempting to add just col_1 to see if col_2 can be defaulted

table = TestDynTbl(name='my_table', description='test table')

❌ This is not possible

table.add_row(col_1 = 'a')
-------------------------------------------------------------------------------------
# Error
ValueError: row data keys don't match available columns
you supplied 0 extra keys: set()
and were missing 1 keys: {'col_2'}

This is of course possible, but no way to actually use col_2 original default_value=3.14

table.add_row(col_1 = 'a', col_2 = -1)
table.to_dataframe()
-------------------------------------------------------------------------------------
    col_1	col_2
id		
0	a	-1

❌ And adding the optional column col_3 (has default_value=0) to that table cannot "back-add" such default value to the first row

table.add_row(col_1 = 'a', col_2 = -1, col_3 = 1.0)
-------------------------------------------------------------------------------------
# Error
ValueError: column must have the same number of rows as 'id'

Traceback for the above error

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Input In [16], in <cell line: 1>()
----> 1 table.add_row(col_1 = 'a', col_2 = -1, col_3 = 1.0)

File .venv/lib/python3.10/site-packages/hdmf/utils.py:645, in docval.<locals>.dec.<locals>.func_call(*args, **kwargs)
    643 def func_call(*args, **kwargs):
    644     pargs = _check_args(args, kwargs)
--> 645     return func(args[0], **pargs)

File .venv/lib/python3.10/site-packages/hdmf/common/table.py:583, in DynamicTable.add_row(self, **kwargs)
    581         if col['name'] in extra_columns:
    582             if data[col['name']] is not None:
--> 583                 self.add_column(col['name'], col['description'],
    584                                 index=col.get('index', False),
    585                                 table=col.get('table', False),
    586                                 enum=col.get('enum', False),
    587                                 col_cls=col.get('class', VectorData),
    588                                 # Pass through extra keyword arguments for add_column that
    589                                 # subclasses may have added
    590                                 **{k: col[k] for k in col.keys()
    591                                    if k not in DynamicTable.__reserved_colspec_keys})
    592             extra_columns.remove(col['name'])
    594 if extra_columns or missing_columns:

File .venv/lib/python3.10/site-packages/hdmf/utils.py:645, in docval.<locals>.dec.<locals>.func_call(*args, **kwargs)
    643 def func_call(*args, **kwargs):
    644     pargs = _check_args(args, kwargs)
--> 645     return func(args[0], **pargs)

File .venv/lib/python3.10/site-packages/hdmf/common/table.py:812, in DynamicTable.add_column(self, **kwargs)
    809     col = col_index
    811 if len(col) != len(self.id):
--> 812     raise ValueError("column must have the same number of rows as 'id'")
    813 self.__colids[name] = len(self.__df_cols)
    814 self.fields['colnames'] = tuple(list(self.colnames) + [name])

ValueError: column must have the same number of rows as 'id'

Attempt to first initialize with all columns then add_row

table = TestDynTbl(name='my_table', description='test table')
table.add_row(col_1 = 'a', col_2 = -1, col_3 = 3.3, col_4=4.4)
table.to_dataframe()
-------------------------------------------------------------------------------------
	col_1	col_2	col_3	col_4
id				
0	a	-1	3.3	4.4

❌ Adding new row with columns that do not have default_value defined in schema (col_1, col_4) did not work.

table.add_row(col_1 = 'b', col_4=1.1)
-------------------------------------------------------------------------------------
# Error
ValueError: row data keys don't match available columns
you supplied 0 extra keys: set()
and were missing 2 keys: {'col_3', 'col_2'}

❌ Adding new row with the required columns (col_1, col_2) and the non-schema-required column that doesn't have default_value (col_4) did not work.

table.add_row(col_1 = 'b', col_2 = 10.2, col_4=1.1)
-------------------------------------------------------------------------------------
# Error
ValueError: row data keys don't match available columns
you supplied 0 extra keys: set()
and were missing 1 keys: {'col_3'}

System

• Ubuntu 20.04
• python 3.10.5
• hdmf 3.4.7
• pynwb 2.2.0
• pandas 1.4.3
• numpy 1.22.4

[rly]

@tuanpham96 Unfortunately, default_value for datasets is not yet supported by PyNWB and HDMF. This is a bug. I created an issue here: hdmf-dev/hdmf#792

In the meantime, you can create a custom API class that should be able to set default values.

[tuanpham96]

Thanks for letting me know! That's a shame but I do understand it might be complicated when it's not a scalar value.

In the meantime, I was thinking of leaving the default_value inside the generated schema, then read the extensions.yaml when creating the API so I wouldn't have to define the names & docs again, as well as replace default_value with default from extensions.yaml.

We can either add support in the API or remove it from the language.

However, if default_value is removed from datasets, such strategy might break in the future.