Skip to content

Commit 65554dc

Browse files
author
Thinh Nguyen
committed
2 parents 1773e23 + d884f01 commit 65554dc

22 files changed

+2770
-267
lines changed

‎.gitignore

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,6 @@
1+
# User data
2+
.DS_Store
3+
14
# Byte-compiled / optimized / DLL files
25
__pycache__/
36
*.py[cod]

‎Background.md

Lines changed: 0 additions & 73 deletions
This file was deleted.

‎CHANGELOG.md

Lines changed: 37 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -1,20 +1,48 @@
11
# Changelog
22

3-
Observes [Semantic Versioning](https://semver.org/spec/v2.0.0.html) standard and [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) convention.
3+
Observes [Semantic Versioning](https://semver.org/spec/v2.0.0.html) standard and
4+
[Keep a Changelog](https://keepachangelog.com/en/1.0.0/) convention.
45

6+
## [0.1.4] - 2022-07-11
7+
8+
+ Bugfix - Handle case where `spike_depths` data is present.
9+
10+
## [0.1.3] - 2022-06-16
11+
12+
+ Update - Allow for the `precluster_output_dir` attribute to be nullable when no pre-clustering is performed.
13+
14+
## [0.1.2] - 2022-06-09
15+
16+
+ Bugfix - Handle case where `pc_features.npy` does not exist.
17+
18+
## [0.1.1] - 2022-06-01
19+
20+
+ Add - Secondary attributes to `PreClusterParamSteps` table
21+
22+
## [0.1.0] - 2022-05-26
23+
24+
+ Update - Rename module for acute probe insertions from `ephys.py` to `ephys_acute.py`.
25+
+ Add - Module for pre-clustering steps (`ephys_precluster.py`), which is built off of `ephys_acute.py`.
26+
+ Add - Module for chronic probe insertions (`ephys_chronic.py`).
27+
+ Bugfix - Missing `fileTimeSecs` key in SpikeGLX meta file.
28+
+ Update - Move common functions to `element-interface` package.
29+
+ Add - NWB export function
530

631
## [0.1.0b0] - 2021-05-07
7-
### Added
8-
+ First beta release
932

33+
+ Update - First beta release
1034

1135
## [0.1.0a5] - 2021-05-05
12-
### Added
13-
+ Added GitHub Action release process
14-
+ `probe` and `ephys` elements
15-
+ Readers for: `SpikeGLX`, `Open Ephys`, `Kilosort`
16-
+ Probe table supporting: Neuropixels probes 1.0 - 3A, 1.0 - 3B, 2.0 - SS, 2.0 - MS
1736

37+
+ Add - GitHub Action release process
38+
+ Add - `probe` and `ephys` elements
39+
+ Add - Readers for: `SpikeGLX`, `Open Ephys`, `Kilosort`
40+
+ Add - Probe table supporting: Neuropixels probes 1.0 - 3A, 1.0 - 3B, 2.0 - SS, 2.0 - MS
1841

19-
[0.1.0b0]: https://github.com/datajoint/element-array-ephys/compare/0.1.0a5...0.1.0b0
42+
[0.1.4]: https://github.com/datajoint/element-array-ephys/releases/tag/0.1.4
43+
[0.1.3]: https://github.com/datajoint/element-array-ephys/releases/tag/0.1.3
44+
[0.1.2]: https://github.com/datajoint/element-array-ephys/releases/tag/0.1.2
45+
[0.1.1]: https://github.com/datajoint/element-array-ephys/releases/tag/0.1.1
46+
[0.1.0]: https://github.com/datajoint/element-array-ephys/compare/0.1.0b0...0.1.0
47+
[0.1.0b0]: https://github.com/datajoint/element-array-ephys/releases/tag/0.1.0b0
2048
[0.1.0a5]: https://github.com/datajoint/element-array-ephys/releases/tag/0.1.0a5

‎README.md

Lines changed: 127 additions & 38 deletions
Original file line numberDiff line numberDiff line change
@@ -1,82 +1,171 @@
11
# DataJoint Element - Array Electrophysiology Element
2-
DataJoint Element for array electrophysiology.
32

4-
This repository features DataJoint pipeline design for extracellular array electrophysiology,
5-
with ***Neuropixels*** probe and ***kilosort*** spike sorting method.
3+
+ This repository features DataJoint pipeline design for extracellular array electrophysiology,
4+
with Neuropixels probe and Kilosort spike sorting method.
65

7-
The pipeline presented here is not a complete pipeline by itself, but rather a modular
6+
+ The pipeline presented here is not a complete pipeline by itself, but rather a modular
87
design of tables and dependencies specific to the extracellular electrophysiology workflow.
98

10-
This modular pipeline element can be flexibly attached downstream
9+
+ This modular pipeline element can be flexibly attached downstream
1110
to any particular design of experiment session, thus assembling a fully functional
1211
ephys pipeline.
1312

14-
See [Background](Background.md) for the background information and development timeline.
13+
+ See the [Element Array Electrophysiology documentation](https://elements.datajoint.org/description/array_ephys/) for the background information and development timeline.
1514

16-
## The Pipeline Architecture
15+
+ For more information on the DataJoint Elements project, please visit https://elements.datajoint.org. This work is supported by the National Institutes of Health.
1716

18-
![element-array-ephys diagram](images/attached_array_ephys_element.svg)
17+
## Element architecture
1918

20-
As the diagram depicts, the array ephys element starts immediately downstream from ***Session***,
21-
and also requires some notion of ***Location*** as a dependency for ***InsertionLocation***.
19+
`element-array-ephys` is comprised of two schemas, `probe` and `ephys`. To handle
20+
several use cases of this pipeline, we have designed several `ephys` schemas, including
21+
`ephys_acute`, `ephys_chronic`, and `ephys_precluster`.
2222

23-
### The design of probe
23+
+ `ephys_acute` - A probe(s) is inserted into a new location during each session.
2424

25-
+ ***ProbeType*** - a lookup table specifying the type of Neuropixels probe (e.g. "neuropixels 1.0", "neuropixels 2.0 single-shank")
26-
+ ***ProbeType.Electrode*** - all electrode and their properties for a particular probe type
25+
+ `ephys_chronic` - A probe(s) is inserted once and used to record across multiple
26+
sessions.
27+
28+
+ `ephys_precluster` - A probe(s) is inserted into a new location during each session.
29+
And pre-clustering steps (e.g. CatGT) are performed on the data from each probe prior
30+
to Kilosort analysis.
31+
32+
### `ephys_acute` module
33+
![element-array-ephys acute diagram](images/attached_array_ephys_element_acute.svg)
34+
35+
36+
### `ephys_chronic` module
37+
![element-array-ephys chronic diagram](images/attached_array_ephys_element_chronic.svg)
38+
39+
### `ephys_precluster` module
40+
![element-array-ephys precluster diagram](
41+
images/attached_array_ephys_element_precluster.svg)
42+
43+
44+
As the diagram depicts, the array ephys element starts immediately downstream from `Session`,
45+
and also requires some notion of `Location` as a dependency for `InsertionLocation`. We
46+
provide an [example workflow](https://github.com/datajoint/workflow-array-ephys/) with a
47+
[pipeline script](https://github.com/datajoint/workflow-array-ephys/blob/main/workflow_array_ephys/pipeline.py)
48+
that models (a) combining this Element with the corresponding [Element-Session](https://github.com/datajoint/element-session)
49+
, and (b) declaring a `SkullReference` table to provide Location.
50+
51+
## Table descriptions
52+
53+
### Probe & electrodes
54+
55+
The `probe` schema contains information regarding the Neuropixels probe and electrode configuration.
56+
57+
<details>
58+
<summary>Click to expand details</summary>
59+
60+
+ `ProbeType` - a lookup table specifying the type of Neuropixels probe (e.g. "neuropixels 1.0", "neuropixels 2.0 single-shank")
61+
62+
+ `ProbeType.Electrode` - all electrode and their properties for a particular probe type
2763
+ An electrode here refers to one recordable electrode site on the Neuropixels probe (e.g. for Neuropixels 1.0, there are 960 sites per shank)
28-
+ ***Probe*** - record of an actual physical probe, identifiable by some unique ID (e.g. probe's serial number)
29-
+ ***ElectrodeConfig*** - particular electrode configuration to be used for ephys recording
30-
+ ***ElectrodeConfig.Electrode*** - corresponding electrodes in ***ProbeType.Electrode*** that are used for recording in this electrode configuration (e.g. for Neuropixels 1.0 or 2.0, there can be at most 384 electrodes usable for recording per probe)
3164

32-
### Extracellular ephys recording
65+
+ `Probe` - record of an actual physical probe, identifiable by some unique ID (e.g. probe's serial number)
66+
67+
+ `ElectrodeConfig` - particular electrode configuration to be used for ephys recording
68+
69+
+ `ElectrodeConfig.Electrode` - corresponding electrodes in `ProbeType.Electrode` that are used for recording in this electrode configuration (e.g. for Neuropixels 1.0 or 2.0, there can be at most 384 electrodes usable for recording per probe)
70+
71+
</details>
72+
73+
### Extracellular electrophysiology recording
74+
75+
The `ephys` schema stores information regarding the recording from a probe for a given session.
76+
77+
<details>
78+
<summary>Click to expand details</summary>
79+
80+
+ `ProbeInsertion` - a surgical insertion of a probe in the brain. Every experimental session consists of one or more entries in `ProbeInsertion` with a corresponding `InsertionLocation` each
81+
82+
+ `EphysRecording` - each `ProbeInsertion` is accompanied by a corresponding `EphysRecording`, specifying the `ElectrodeConfig` used for the recording from the `Probe` defined in such `ProbeInsertion`
83+
84+
</details>
3385

34-
+ ***ProbeInsertion*** - a surgical insertion of a probe in the brain. Every experimental session consists of one or more entries in ***ProbeInsertion*** with a corresponding ***InsertionLocation*** each
35-
+ ***EphysRecording*** - each ***ProbeInsertion*** is accompanied by a corresponding ***EphysRecording***, specifying the ***ElectrodeConfig*** used for the recording from the ***Probe*** defined in such ***ProbeInsertion***
36-
3786
### Clusters and spikes
3887

39-
This ephys element features automatic ingestion for spike sorting results from the ***kilosort*** method.
88+
The `ephys` schema features automatic ingestion of spike sorting results from the `Kilosort` analysis method.
89+
90+
<details>
91+
<summary>Click to expand details</summary>
92+
93+
+ `Clustering` - specify instance(s) of clustering on an `EphysRecording`, by some `ClusteringMethod`
4094

41-
+ ***Clustering*** - specify instance(s) of clustering on an ***EphysRecording***, by some ***ClusteringMethod***
42-
+ ***Curation*** - specify instance(s) of curations performed on the output of a given ***Clustering***
43-
+ ***CuratedClustering*** - set of results from a particular round of clustering/curation
44-
+ ***CuratedClustering.Unit*** - Identified unit(s) from one ***Curation***, and the associated properties (e.g. cluster quality, spike times, spike depths, etc.)
45-
+ ***WaveformSet*** - A set of spike waveforms for units from a given CuratedClustering
95+
+ `Curation` - specify instance(s) of curations performed on the output of a given `Clustering`
96+
97+
+ `CuratedClustering` - set of results from a particular round of clustering/curation
98+
+ `CuratedClustering.Unit` - Identified unit(s) from one `Curation`, and the associated properties (e.g. cluster quality, spike times, spike depths, etc.)
99+
+ `WaveformSet` - A set of spike waveforms for units from a given CuratedClustering
100+
101+
</details>
46102

47103
## Installation
48-
```
49-
pip install element-array-ephys
50-
```
104+
<details>
105+
<summary>Click to expand details</summary>
106+
107+
+ Install `element-array-ephys`
108+
```
109+
pip install element-array-ephys
110+
```
111+
112+
+ Upgrade `element-array-ephys` previously installed with `pip`
113+
```
114+
pip install --upgrade element-array-ephys
115+
```
116+
117+
+ Install `element-interface`
118+
+ `element-interface` is a dependency of `element-array-ephys`, however it is not contained within `requirements.txt`.
119+
```
120+
pip install "element-interface @ git+https://github.com/datajoint/element-interface"
121+
```
51122
52-
If you already have an older version of ***element-array-ephys*** installed using `pip`, upgrade with
53-
```
54-
pip install --upgrade element-array-ephys
55-
```
123+
</details>
56124
57125
## Usage
58126
59127
### Element activation
60128
129+
When using this Element, one needs to run `ephys.activate` to declare the schemas and tables on the database.
130+
131+
<details>
132+
<summary>Click to expand details</summary>
133+
61134
To activate the `element-array-ephys`, ones need to provide:
62135
63136
1. Schema names
64137
+ schema name for the probe module
65138
+ schema name for the ephys module
66139
67140
2. Upstream tables
68-
+ Session table
69-
+ SkullReference table (Reference table for InsertionLocation, specifying the skull reference)
141+
+ Session table: A set of keys identifying a recording session (see [Element-Session](https://github.com/datajoint/element-session)).
142+
+ SkullReference table: A reference table for InsertionLocation, specifying the skull reference (see [example pipeline](https://github.com/datajoint/workflow-array-ephys/blob/main/workflow_array_ephys/pipeline.py)).
70143
71-
3. Utility functions
72-
+ get_ephys_root_data_dir()
73-
+ get_session_directory()
144+
3. Utility functions. See [example definitions here](https://github.com/datajoint/workflow-array-ephys/blob/main/workflow_array_ephys/paths.py)
145+
+ get_ephys_root_data_dir(): Returns your root data directory.
146+
+ get_session_directory(): Returns the path of the session data relative to the root.
74147
75148
For more detail, check the docstring of the `element-array-ephys`:
76149
77150
help(probe.activate)
78151
help(ephys.activate)
79152
153+
</details>
154+
80155
### Example usage
81156
82157
See [this project](https://github.com/datajoint/workflow-array-ephys) for an example usage of this Array Electrophysiology Element.
158+
159+
## Citation
160+
161+
+ If your work uses DataJoint and DataJoint Elements, please cite the respective Research Resource Identifiers (RRIDs) and manuscripts.
162+
163+
+ DataJoint for Python or MATLAB
164+
+ Yatsenko D, Reimer J, Ecker AS, Walker EY, Sinz F, Berens P, Hoenselaar A, Cotton RJ, Siapas AS, Tolias AS. DataJoint: managing big scientific data using MATLAB or Python. bioRxiv. 2015 Jan 1:031658. doi: https://doi.org/10.1101/031658
165+
166+
+ DataJoint ([RRID:SCR_014543](https://scicrunch.org/resolver/SCR_014543)) - DataJoint for `<Select Python or MATLAB>` (version `<Enter version number>`)
167+
168+
+ DataJoint Elements
169+
+ Yatsenko D, Nguyen T, Shen S, Gunalan K, Turner CA, Guzman R, Sasaki M, Sitonic D, Reimer J, Walker EY, Tolias AS. DataJoint Elements: Data Workflows for Neurophysiology. bioRxiv. 2021 Jan 1. doi: https://doi.org/10.1101/2021.03.30.437358
170+
171+
+ DataJoint Elements ([RRID:SCR_021894](https://scicrunch.org/resolver/SCR_021894)) - Element Array Electrophysiology (version `<Enter version number>`)

0 commit comments

Comments
 (0)