Skip to content

Commit ddd4095

Browse files
author
Thinh Nguyen
committed
Merge remote-tracking branch 'upstream/run_kilosort' into no-curation
2 parents 7fecff1 + 7a4fba9 commit ddd4095

File tree

8 files changed

+777
-102
lines changed

8 files changed

+777
-102
lines changed

‎Background.md

Lines changed: 0 additions & 73 deletions
This file was deleted.

‎README.md

Lines changed: 79 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -1,53 +1,85 @@
11
# DataJoint Element - Array Electrophysiology Element
22

3-
This repository features DataJoint pipeline design for extracellular array electrophysiology,
4-
with ***Neuropixels*** probe and ***kilosort*** spike sorting method.
3+
+ This repository features DataJoint pipeline design for extracellular array electrophysiology,
4+
with Neuropixels probe and Kilosort spike sorting method.
55

6-
The pipeline presented here is not a complete pipeline by itself, but rather a modular
6+
+ The pipeline presented here is not a complete pipeline by itself, but rather a modular
77
design of tables and dependencies specific to the extracellular electrophysiology workflow.
88

9-
This modular pipeline element can be flexibly attached downstream
9+
+ This modular pipeline element can be flexibly attached downstream
1010
to any particular design of experiment session, thus assembling a fully functional
1111
ephys pipeline.
1212

13-
See [Background](Background.md) for the background information and development timeline.
13+
+ See the [Element Array Electrophysiology documentation](https://elements.datajoint.org/description/array_ephys/) for the background information and development timeline.
14+
15+
+ For more information on the DataJoint Elements project, please visit https://elements.datajoint.org. This work is supported by the National Institutes of Health.
1416

1517
## Element architecture
1618

1719
![element-array-ephys diagram](images/attached_array_ephys_element.svg)
1820

19-
As the diagram depicts, the array ephys element starts immediately downstream from ***Session***,
20-
and also requires some notion of ***Location*** as a dependency for ***InsertionLocation***. We
21+
As the diagram depicts, the array ephys element starts immediately downstream from `Session`,
22+
and also requires some notion of `Location` as a dependency for `InsertionLocation`. We
2123
provide an [example workflow](https://github.com/datajoint/workflow-array-ephys/) with a
2224
[pipeline script](https://github.com/datajoint/workflow-array-ephys/blob/main/workflow_array_ephys/pipeline.py)
2325
that models (a) combining this Element with the corresponding [Element-Session](https://github.com/datajoint/element-session)
24-
, and (b) declaring a ***SkullReference*** table to provide Location.
26+
, and (b) declaring a `SkullReference` table to provide Location.
27+
28+
## Table descriptions
29+
30+
### Probe & electrodes
31+
32+
The `probe` schema contains information regarding the Neuropixels probe and electrode configuration.
2533

26-
### The design of probe
34+
<details>
35+
<summary>Click to expand details</summary>
2736

28-
+ ***ProbeType*** - a lookup table specifying the type of Neuropixels probe (e.g. "neuropixels 1.0", "neuropixels 2.0 single-shank")
29-
+ ***ProbeType.Electrode*** - all electrode and their properties for a particular probe type
37+
+ `ProbeType` - a lookup table specifying the type of Neuropixels probe (e.g. "neuropixels 1.0", "neuropixels 2.0 single-shank")
38+
39+
+ `ProbeType.Electrode` - all electrode and their properties for a particular probe type
3040
+ An electrode here refers to one recordable electrode site on the Neuropixels probe (e.g. for Neuropixels 1.0, there are 960 sites per shank)
31-
+ ***Probe*** - record of an actual physical probe, identifiable by some unique ID (e.g. probe's serial number)
32-
+ ***ElectrodeConfig*** - particular electrode configuration to be used for ephys recording
33-
+ ***ElectrodeConfig.Electrode*** - corresponding electrodes in ***ProbeType.Electrode*** that are used for recording in this electrode configuration (e.g. for Neuropixels 1.0 or 2.0, there can be at most 384 electrodes usable for recording per probe)
3441

35-
### Extracellular ephys recording
42+
+ `Probe` - record of an actual physical probe, identifiable by some unique ID (e.g. probe's serial number)
43+
44+
+ `ElectrodeConfig` - particular electrode configuration to be used for ephys recording
45+
46+
+ `ElectrodeConfig.Electrode` - corresponding electrodes in `ProbeType.Electrode` that are used for recording in this electrode configuration (e.g. for Neuropixels 1.0 or 2.0, there can be at most 384 electrodes usable for recording per probe)
47+
48+
</details>
49+
50+
### Extracellular electrophysiology recording
51+
52+
The `ephys` schema stores information regarding the recording from a probe for a given session.
53+
54+
<details>
55+
<summary>Click to expand details</summary>
56+
57+
+ `ProbeInsertion` - a surgical insertion of a probe in the brain. Every experimental session consists of one or more entries in `ProbeInsertion` with a corresponding `InsertionLocation` each
58+
59+
+ `EphysRecording` - each `ProbeInsertion` is accompanied by a corresponding `EphysRecording`, specifying the `ElectrodeConfig` used for the recording from the `Probe` defined in such `ProbeInsertion`
60+
61+
</details>
3662

37-
+ ***ProbeInsertion*** - a surgical insertion of a probe in the brain. Every experimental session consists of one or more entries in ***ProbeInsertion*** with a corresponding ***InsertionLocation*** each
38-
+ ***EphysRecording*** - each ***ProbeInsertion*** is accompanied by a corresponding ***EphysRecording***, specifying the ***ElectrodeConfig*** used for the recording from the ***Probe*** defined in such ***ProbeInsertion***
39-
4063
### Clusters and spikes
4164

42-
This ephys element features automatic ingestion for spike sorting results from the ***kilosort*** method.
65+
The `ephys` schema features automatic ingestion of spike sorting results from the `Kilosort` analysis method.
66+
67+
<details>
68+
<summary>Click to expand details</summary>
4369

44-
+ ***Clustering*** - specify instance(s) of clustering on an ***EphysRecording***, by some ***ClusteringMethod***
45-
+ ***Curation*** - specify instance(s) of curations performed on the output of a given ***Clustering***
46-
+ ***CuratedClustering*** - set of results from a particular round of clustering/curation
47-
+ ***CuratedClustering.Unit*** - Identified unit(s) from one ***Curation***, and the associated properties (e.g. cluster quality, spike times, spike depths, etc.)
48-
+ ***WaveformSet*** - A set of spike waveforms for units from a given CuratedClustering
70+
+ `Clustering` - specify instance(s) of clustering on an `EphysRecording`, by some `ClusteringMethod`
71+
72+
+ `Curation` - specify instance(s) of curations performed on the output of a given `Clustering`
73+
74+
+ `CuratedClustering` - set of results from a particular round of clustering/curation
75+
+ `CuratedClustering.Unit` - Identified unit(s) from one `Curation`, and the associated properties (e.g. cluster quality, spike times, spike depths, etc.)
76+
+ `WaveformSet` - A set of spike waveforms for units from a given CuratedClustering
77+
78+
</details>
4979

5080
## Installation
81+
<details>
82+
<summary>Click to expand details</summary>
5183

5284
+ Install `element-array-ephys`
5385
```
@@ -60,17 +92,22 @@ This ephys element features automatic ingestion for spike sorting results from t
6092
```
6193
6294
+ Install `element-interface`
63-
6495
+ `element-interface` is a dependency of `element-array-ephys`, however it is not contained within `requirements.txt`.
65-
6696
```
6797
pip install "element-interface @ git+https://github.com/datajoint/element-interface"
6898
```
6999
100+
</details>
101+
70102
## Usage
71103
72104
### Element activation
73105
106+
When using this Element, one needs to run `ephys.activate` to declare the schemas and tables on the database.
107+
108+
<details>
109+
<summary>Click to expand details</summary>
110+
74111
To activate the `element-array-ephys`, ones need to provide:
75112
76113
1. Schema names
@@ -90,6 +127,22 @@ For more detail, check the docstring of the `element-array-ephys`:
90127
help(probe.activate)
91128
help(ephys.activate)
92129
130+
</details>
131+
93132
### Example usage
94133
95134
See [this project](https://github.com/datajoint/workflow-array-ephys) for an example usage of this Array Electrophysiology Element.
135+
136+
## Citation
137+
138+
+ If your work uses DataJoint and DataJoint Elements, please cite the respective Research Resource Identifiers (RRIDs) and manuscripts.
139+
140+
+ DataJoint for Python or MATLAB
141+
+ Yatsenko D, Reimer J, Ecker AS, Walker EY, Sinz F, Berens P, Hoenselaar A, Cotton RJ, Siapas AS, Tolias AS. DataJoint: managing big scientific data using MATLAB or Python. bioRxiv. 2015 Jan 1:031658. doi: https://doi.org/10.1101/031658
142+
143+
+ DataJoint ([RRID:SCR_014543](https://scicrunch.org/resolver/SCR_014543)) - DataJoint for `<Select Python or MATLAB>` (version `<Enter version number>`)
144+
145+
+ DataJoint Elements
146+
+ Yatsenko D, Nguyen T, Shen S, Gunalan K, Turner CA, Guzman R, Sasaki M, Sitonic D, Reimer J, Walker EY, Tolias AS. DataJoint Elements: Data Workflows for Neurophysiology. bioRxiv. 2021 Jan 1. doi: https://doi.org/10.1101/2021.03.30.437358
147+
148+
+ DataJoint Elements ([RRID:SCR_021894](https://scicrunch.org/resolver/SCR_021894)) - Element Array Electrophysiology (version `<Enter version number>`)

‎element_array_ephys/ephys_chronic.py

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -155,7 +155,7 @@ class EphysRecording(dj.Imported):
155155
---
156156
-> probe.ElectrodeConfig
157157
-> AcquisitionSoftware
158-
sampling_rate: float # (Hz)
158+
sampling_rate: float # (Hz)
159159
recording_datetime: datetime # datetime of the recording from this probe
160160
recording_duration: float # (seconds) duration of the recording from this probe
161161
"""
@@ -195,7 +195,8 @@ def make(self, key):
195195
break
196196
else:
197197
raise FileNotFoundError(
198-
'No SpikeGLX data found for probe insertion: {}'.format(key))
198+
f'No SpikeGLX data found for probe insertion: {key}' +
199+
' The probe serial number does not match.')
199200

200201
if spikeglx_meta.probe_model in supported_probe_types:
201202
probe_type = spikeglx_meta.probe_model

‎element_array_ephys/export/nwb/README.md

Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
# Exporting data to NWB
2+
3+
The `export/nwb/nwb.py` module maps from the element-array-ephys data structure to NWB.
4+
The main function is `ecephys_session_to_nwb`, which contains flags to control calling the following functions,
5+
which can be called independently:
6+
7+
1. `session.export.nwb.session_to_nwb`: Gathers session-level metadata
8+
9+
10+
2. `add_electrodes_to_nwb`: Add electrodes table to NWBFile. This is needed for any ElectricalSeries, including
11+
raw source data and LFP.
12+
13+
14+
ephys.InsertionLocation -> ElectrodeGroup.location
15+
16+
probe.Probe::probe -> device.name
17+
probe.Probe::probe_comment -> device.description
18+
probe.Probe::probe_type -> device.manufacturer
19+
20+
probe.ProbeType.Electrode::electrode -> electrodes["id_in_probe"]
21+
probe.ProbeType.Electrode::y_coord -> electrodes["rel_y"]
22+
probe.ProbeType.Electrode::x_coord -> electrodes["rel_x"]
23+
probe.ProbeType.Electrode::shank -> electrodes["shank"]
24+
probe.ProbeType.Electrode::shank_col -> electrodes["shank_col"]
25+
probe.ProbeType.Electrode::shank_row -> electrodes["shank_row"]
26+
27+
3. `add_ephys_recording_to_nwb`: Read voltage data directly from source files and iteratively transfer them to the
28+
NWB file. Automatically applies lossless compression to the data, so the final file might be smaller than the original, but there is no
29+
data loss. Currently supports Neuropixels data acquired with SpikeGLX or Open Ephys, and relies on SpikeInterface to read the data.
30+
31+
32+
source data -> acquisition["ElectricalSeries"]
33+
34+
4. `add_ephys_units_to_nwb`: Add spiking data from CuratedClustering to NWBFile.
35+
36+
37+
ephys.CuratedClustering.Unit::unit -> units.id
38+
ephys.CuratedClustering.Unit::spike_times -> units["spike_times"]
39+
ephys.CuratedClustering.Unit::spike_depths -> units["spike_depths"]
40+
ephys.CuratedClustering.Unit::cluster_quality_label -> units["cluster_quality_label"]
41+
42+
ephys.WaveformSet.PeakWaveform::peak_electrode_waveform -> units["waveform_mean"]
43+
44+
5. `add_ephys_lfp_from_dj_to_nwb`: Read LFP data from the data in element-array-ephys and convert to NWB.
45+
46+
47+
ephys.LFP.Electrode::lfp -> processing["ecephys"].lfp.electrical_series["ElectricalSeries{insertion_number}"].data
48+
ephys.LFP::lfp_time_stamps -> processing["ecephys"].lfp.electrical_series["ElectricalSeries{insertion_number}"].timestamps
49+
50+
6. `add_ephys_lfp_from_source_to_nwb`: Read the LFP data directly from the source file. Currently, only works for
51+
SpikeGLX data. Can be used instead of `add_ephys_lfp_from_dj_to_nwb`.
52+
53+
54+
source data -> processing["ecephys"].lfp.electrical_series["ElectricalSeries{insertion_number}"].data
55+
source data -> processing["ecephys"].lfp.electrical_series["ElectricalSeries{insertion_number}"].timestamps

‎element_array_ephys/export/nwb/__init__.py

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
from .nwb import ecephys_session_to_nwb, write_nwb

0 commit comments

Comments
 (0)