Merge branch 'main' of https://github.com/datajoint/element-array-ephys

Author: CBroz1

.gitignore

@@ -1,3 +1,6 @@
+# User data
+.DS_Store
+
 # Byte-compiled / optimized / DLL files
 __pycache__/
 *.py[cod]

Background.md

@@ -1,82 +1,148 @@
 # DataJoint Element - Array Electrophysiology Element
-DataJoint Element for array electrophysiology.
 
-This repository features DataJoint pipeline design for extracellular array electrophysiology, 
-with ***Neuropixels*** probe and ***kilosort*** spike sorting method. 
++ This repository features DataJoint pipeline design for extracellular array electrophysiology, 
+with Neuropixels probe and Kilosort spike sorting method. 
 
-The pipeline presented here is not a complete pipeline by itself, but rather a modular 
++ The pipeline presented here is not a complete pipeline by itself, but rather a modular 
 design of tables and dependencies specific to the extracellular electrophysiology workflow. 
 
-This modular pipeline element can be flexibly attached downstream 
++ This modular pipeline element can be flexibly attached downstream 
 to any particular design of experiment session, thus assembling a fully functional 
 ephys pipeline.
 
-See [Background](Background.md) for the background information and development timeline.
++ See the [Element Array Electrophysiology documentation](https://elements.datajoint.org/description/array_ephys/) for the background information and development timeline.
 
-## The Pipeline Architecture
++ For more information on the DataJoint Elements project, please visit https://elements.datajoint.org.  This work is supported by the National Institutes of Health.
+
+## Element architecture
 
 ![element-array-ephys diagram](images/attached_array_ephys_element.svg)
 
-As the diagram depicts, the array ephys element starts immediately downstream from ***Session***, 
-and also requires some notion of ***Location*** as a dependency for ***InsertionLocation***.
+As the diagram depicts, the array ephys element starts immediately downstream from `Session`, 
+and also requires some notion of `Location` as a dependency for `InsertionLocation`. We 
+provide an [example workflow](https://github.com/datajoint/workflow-array-ephys/) with a 
+[pipeline script](https://github.com/datajoint/workflow-array-ephys/blob/main/workflow_array_ephys/pipeline.py)
+that models (a) combining this Element with the corresponding [Element-Session](https://github.com/datajoint/element-session)
+, and (b) declaring a `SkullReference` table to provide Location.
+
+## Table descriptions
+
+### Probe & electrodes
+
+The `probe` schema contains information regarding the Neuropixels probe and electrode configuration.
 
-### The design of probe
+<details>
+<summary>Click to expand details</summary>
 
-+ ***ProbeType*** - a lookup table specifying the type of Neuropixels probe (e.g. "neuropixels 1.0", "neuropixels 2.0 single-shank")
-+ ***ProbeType.Electrode*** - all electrode and their properties for a particular probe type
++ `ProbeType` - a lookup table specifying the type of Neuropixels probe (e.g. "neuropixels 1.0", "neuropixels 2.0 single-shank")
+
++ `ProbeType.Electrode` - all electrode and their properties for a particular probe type
     + An electrode here refers to one recordable electrode site on the Neuropixels probe (e.g. for Neuropixels 1.0, there are 960 sites per shank)
-+ ***Probe*** - record of an actual physical probe, identifiable by some unique ID (e.g. probe's serial number)
-+ ***ElectrodeConfig*** - particular electrode configuration to be used for ephys recording
-+ ***ElectrodeConfig.Electrode*** - corresponding electrodes in ***ProbeType.Electrode*** that are used for recording in this electrode configuration (e.g. for Neuropixels 1.0 or 2.0, there can be at most 384 electrodes usable for recording per probe)
 
-### Extracellular ephys recording
++ `Probe` - record of an actual physical probe, identifiable by some unique ID (e.g. probe's serial number)
+
++ `ElectrodeConfig` - particular electrode configuration to be used for ephys recording
+
++ `ElectrodeConfig.Electrode` - corresponding electrodes in `ProbeType.Electrode` that are used for recording in this electrode configuration (e.g. for Neuropixels 1.0 or 2.0, there can be at most 384 electrodes usable for recording per probe)
+
+</details>
+
+### Extracellular electrophysiology recording
+
+The `ephys` schema stores information regarding the recording from a probe for a given session.
+
+<details>
+<summary>Click to expand details</summary>
+
++ `ProbeInsertion` - a surgical insertion of a probe in the brain. Every experimental session consists of one or more entries in `ProbeInsertion` with a corresponding `InsertionLocation` each
+
++ `EphysRecording` - each `ProbeInsertion` is accompanied by a corresponding `EphysRecording`, specifying the `ElectrodeConfig` used for the recording from the `Probe` defined in such `ProbeInsertion`
+
+</details>
 
-+ ***ProbeInsertion*** - a surgical insertion of a probe in the brain. Every experimental session consists of one or more entries in ***ProbeInsertion*** with a corresponding ***InsertionLocation*** each
-+ ***EphysRecording*** - each ***ProbeInsertion*** is accompanied by a corresponding ***EphysRecording***, specifying the ***ElectrodeConfig*** used for the recording from the ***Probe*** defined in such ***ProbeInsertion***
-    
 ### Clusters and spikes
 
-This ephys element features automatic ingestion for spike sorting results from the ***kilosort*** method. 
+The `ephys` schema features automatic ingestion of spike sorting results from the `Kilosort` analysis method. 
+
+<details>
+<summary>Click to expand details</summary>
 
-+ ***Clustering*** - specify instance(s) of clustering on an ***EphysRecording***, by some ***ClusteringMethod***
-+ ***Curation*** - specify instance(s) of curations performed on the output of a given ***Clustering***
-+ ***CuratedClustering*** - set of results from a particular round of clustering/curation
-    + ***CuratedClustering.Unit*** - Identified unit(s) from one ***Curation***, and the associated properties (e.g. cluster quality, spike times, spike depths, etc.)
-    + ***WaveformSet*** - A set of spike waveforms for units from a given CuratedClustering
++ `Clustering` - specify instance(s) of clustering on an `EphysRecording`, by some `ClusteringMethod`
+
++ `Curation` - specify instance(s) of curations performed on the output of a given `Clustering`
+
++ `CuratedClustering` - set of results from a particular round of clustering/curation
+    + `CuratedClustering.Unit` - Identified unit(s) from one `Curation`, and the associated properties (e.g. cluster quality, spike times, spike depths, etc.)
+    + `WaveformSet` - A set of spike waveforms for units from a given CuratedClustering
+
+</details>
 
 ## Installation
-```
-pip install element-array-ephys
-```
+<details>
+<summary>Click to expand details</summary>
+
++ Install `element-array-ephys`
+    ```
+    pip install element-array-ephys
+    ```
 
-If you already have an older version of ***element-array-ephys*** installed using `pip`, upgrade with
-```
-pip install --upgrade element-array-ephys
-```
++ Upgrade `element-array-ephys` previously installed with `pip`
+    ```
+    pip install --upgrade element-array-ephys
+    ```
+
++ Install `element-interface`
+    + `element-interface` is a dependency of `element-array-ephys`, however it is not contained within `requirements.txt`.
+    ```
+    pip install "element-interface @ git+https://github.com/datajoint/element-interface"
+    ```
+
+</details>
 
 ## Usage
 
 ### Element activation
 
+When using this Element, one needs to run `ephys.activate` to declare the schemas and tables on the database.
+
+<details>
+<summary>Click to expand details</summary>
+
 To activate the `element-array-ephys`, ones need to provide:
 
 1. Schema names
     + schema name for the probe module
     + schema name for the ephys module
 
 2. Upstream tables
-    + Session table
-    + SkullReference table (Reference table for InsertionLocation, specifying the skull reference)
+    + Session table: A set of keys identifying a recording session (see [Element-Session](https://github.com/datajoint/element-session)).
+    + SkullReference table: A reference table for InsertionLocation, specifying the skull reference (see [example pipeline](https://github.com/datajoint/workflow-array-ephys/blob/main/workflow_array_ephys/pipeline.py)).
 
-3. Utility functions
-    + get_ephys_root_data_dir()
-    + get_session_directory()
+3. Utility functions. See [example definitions here](https://github.com/datajoint/workflow-array-ephys/blob/main/workflow_array_ephys/paths.py)
+    + get_ephys_root_data_dir(): Returns your root data directory.
+    + get_session_directory(): Returns the path of the session data relative to the root.
 
 For more detail, check the docstring of the `element-array-ephys`:
 
     help(probe.activate)
     help(ephys.activate)
 
+</details>
+
 ### Example usage
 
 See [this project](https://github.com/datajoint/workflow-array-ephys) for an example usage of this Array Electrophysiology Element.
+
+## Citation
+
++ If your work uses DataJoint and DataJoint Elements, please cite the respective Research Resource Identifiers (RRIDs) and manuscripts.
+
++ DataJoint for Python or MATLAB
+    + Yatsenko D, Reimer J, Ecker AS, Walker EY, Sinz F, Berens P, Hoenselaar A, Cotton RJ, Siapas AS, Tolias AS. DataJoint: managing big scientific data using MATLAB or Python. bioRxiv. 2015 Jan 1:031658. doi: https://doi.org/10.1101/031658
+
+    + DataJoint ([RRID:SCR_014543](https://scicrunch.org/resolver/SCR_014543)) - DataJoint for `<Select Python or MATLAB>` (version `<Enter version number>`)
+
++ DataJoint Elements
+    + Yatsenko D, Nguyen T, Shen S, Gunalan K, Turner CA, Guzman R, Sasaki M, Sitonic D, Reimer J, Walker EY, Tolias AS. DataJoint Elements: Data Workflows for Neurophysiology. bioRxiv. 2021 Jan 1. doi: https://doi.org/10.1101/2021.03.30.437358
+
+    + DataJoint Elements ([RRID:SCR_021894](https://scicrunch.org/resolver/SCR_021894)) - Element Array Electrophysiology (version `<Enter version number>`)

README.md

@@ -1,69 +1,2 @@
-import datajoint as dj
-import pathlib
-import uuid
-import hashlib
-
-
-dj.config['enable_python_native_blobs'] = True
-
-
-def find_full_path(root_directories, relative_path):
-    """
-    Given a relative path, search and return the full-path
-     from provided potential root directories (in the given order)
-        :param root_directories: potential root directories
-        :param relative_path: the relative path to find the valid root directory
-        :return: root_directory (pathlib.Path object)
-    """
-    relative_path = pathlib.Path(relative_path)
-
-    if relative_path.exists():
-        return relative_path
-
-    # turn to list if only a single root directory is provided
-    if isinstance(root_directories, (str, pathlib.Path)):
-        root_directories = [root_directories]
-
-    for root_dir in root_directories:
-        if (pathlib.Path(root_dir) / relative_path).exists():
-            return pathlib.Path(root_dir) / relative_path
-
-    raise FileNotFoundError('No valid full-path found (from {})'
-                            ' for {}'.format(root_directories, relative_path))
-
-
-def find_root_directory(root_directories, full_path):
-    """
-    Given multiple potential root directories and a full-path,
-    search and return one directory that is the parent of the given path
-        :param root_directories: potential root directories
-        :param full_path: the relative path to search the root directory
-        :return: full-path (pathlib.Path object)
-    """
-    full_path = pathlib.Path(full_path)
-
-    if not full_path.exists():
-        raise FileNotFoundError(f'{full_path} does not exist!')
-
-    # turn to list if only a single root directory is provided
-    if isinstance(root_directories, (str, pathlib.Path)):
-        root_directories = [root_directories]
-
-    try:
-        return next(pathlib.Path(root_dir) for root_dir in root_directories
-                    if pathlib.Path(root_dir) in set(full_path.parents))
-
-    except StopIteration:
-        raise FileNotFoundError('No valid root directory found (from {})'
-                                ' for {}'.format(root_directories, full_path))
-
-
-def dict_to_uuid(key):
-    """
-    Given a dictionary `key`, returns a hash string as UUID
-    """
-    hashed = hashlib.md5()
-    for k, v in sorted(key.items()):
-        hashed.update(str(k).encode())
-        hashed.update(str(v).encode())
-    return uuid.UUID(hex=hashed.hexdigest())
+# ephys_acute as default
+import element_array_ephys.ephys_acute as ephys