Porting PyNWB to JavaScript

[garrettmflynn]

I've run some successful tests for reading/writing HDF5 files in the browser (see jsnwb). However, I'm running into issues porting the PyNWB code since it has extensive dependencies on the related HDMF project.

At the end of the day, the goal is to unfold the HDF5 file into an NWBFile object This issue might already be well-documented, but could I get some guidance on how this was handled in the MatNWB library? As well as how the nwb-schema library should fit into all this?

• I've read enough to understand that MatNWB generates classes directly from the schema while PyNWB uses HDMF which allows more flexibility (e.g. converting between versions). Though I'm not sure what the best approach would be for a new language port.

To demonstrate where the project is at, I've published an demo site that loads an example NWB file (or one supplied by the user), deconstructs the HDF5 hierarchy, and prints this to a terminal-like container on the screen.

[bendichter]

@garrettmflynn wow, very cool! Yes, I think the MatNWB approach is the best approach for you here. Another very important question is whether you want to support just reading or reading and writing. Supporting writing is more difficult because you must

1. enforce the schema
2. allow for advanced write features like chunking, compression, and iterative write. This is a must for a writing API, as it is very common for NWB files to be < 10 GB

If all you really need is read, I think it would be a good idea to limit to that scope up-front and make your life a lot easier.

@ln-vidrio is the lead architect of MatNWB, so I think talking to him would be important.

For reading, you already have an HDF5 reader, so basically the question is how you could make reading NWB files syntactically easier. In Python, for instance, we allow users to convert DynamicTables into pandas DataFrames. I think it might be useful to do something similar in JS, converting to common data types in that language, but I don't know enough about JS to know what those types would be.

[oruebel]

I agree with Ben. For JS, limiting (at least initially) the scope to supporting read will make things much easier. If the goal is a pure JS library (i.e., without calls to Python), then I think the approach of auto-generating the API from the schema will be the fastest way to get to a fully functional API that will also support extensions. Since the JS API will most likely run in the browser, I think a dynamic generation of objects (similar to how PyNWB supports extensions) could potentially be simpler than a static code generation, as it would keep the library small, avoids having to generate physical .js files for different versions of the NWB schema, and could make it easier to deal with extensions. Since you are already able to read the HDF5 file itself, the main question really becomes what level of additional convenience functions you want the API to support and how closely it can/should match PyNWB or MatNWB. I can't wait to learn more about this project at the upcoming developer hackathon.

[garrettmflynn]

Thank you @bendichter and @oruebel for your feedback! I'm also in conversation with the Neurodata without Borders Explorer team to see where this tool might fit into their ecosystem.

@oruebel Where should I reference to get the best explanation/demonstration of the PyNWB dynamic object generation for extension support? I'm currently generating simple objects based on the NWB file structure when reading—but obviously having the top-down guidance of the schema would make the creation of convenience function more, well, convenient.

[oruebel]

Where should I reference to get the best explanation/demonstration of the PyNWB dynamic object generation ...

In PyNWB, the get_class method is defined here. However this mostly a convenient wrapper for TypeMap.get_dt_container_cls. The majority of the actual code to generate classes in Python is then defined in hdmf.build.classgenerator. @rly may be more familiar with details of the class generator in Python than I am.

As @bendichter mentioned, @ln-vidrio will also be a good person to talk to, as he is most familiar with the class generation in Matlab.

[lawrence-mbf]

@garrettmflynn I've not much to add but I am watching this thread.

In MatNWB, you will find most class generation functions in the +fill module whereas class parsing is mostly spread around the schemes.Namespace class and spec module. Feel free to ask if you wish dig into MatNWB.