Description
Feature Request
Problem Statement
While the database provides data structure, efficient queries, and transaction support, files are still preferred for strong large objects such as images, numerical arrays, movies, etc. Users like to have direct read-only access to the files without mediation by the database. Storing large objects in MySQL tables has adverse performance effects on data queries.
DataJoint has previously implemented several approaches to address some aspects of this problem:
- Storing file paths as varchar strings with the user responsible for the file management.
- The
attach
andattach@store
datatype to store files, preserving the filename but not the folder structures - The
blob@store
datatype for storing serialized data structures in external files - The
filepath@store
datatype to allow organizing files and folders under users' control - The
AdapatedType
datatype that allows defining custom logic to apply for reading and writing.
In particular, the SpyGlass pipeline Loren Frank's lab relied on the filepath
and AdaptedType
features to implement NWB file management.
None of these methods simultaneously address the following desiderata:
- A logical, consistent file folder structure that's prescribed by DataJoint, based on the schema design and primary key values
- Keeping files in their original form and extension so that they can read and used outside DataJoint. Files should be accessible for reading without
datajoint-python
or DB access, and files should maintain their native file extensions and MIME types (as opposed to serializing into another format). - Files are copied into their location and referenced in a single step as part of the insert and fetch operations.
- Files are deleted when the table entries referencing them are deleted
- Data consistency through transaction processing: inserts and deletes are executed as atomic transactions that can rollback when the transaction fails and where concurrent transactions do not lead to inconsistencies.
We need a solution for file management that simultaneously addresses all of these desiderata.