diff --git a/.github/workflows/test.yaml b/.github/workflows/test.yaml index a463737f..36199ef3 100644 --- a/.github/workflows/test.yaml +++ b/.github/workflows/test.yaml @@ -8,8 +8,6 @@ on: - main workflow_dispatch: jobs: - devcontainer-build: - uses: datajoint/.github/.github/workflows/devcontainer-build.yaml@main tests: runs-on: ubuntu-latest strategy: @@ -30,3 +28,4 @@ jobs: run: | python_version=${{matrix.py_ver}} black element_array_ephys --check --verbose --target-version py${python_version//.} + black notebooks --check --verbose --target-version py${python_version//.} diff --git a/element_array_ephys/plotting/corr.py b/element_array_ephys/plotting/corr.py index a8daa8cb..d94a0d9f 100644 --- a/element_array_ephys/plotting/corr.py +++ b/element_array_ephys/plotting/corr.py @@ -1,5 +1,4 @@ -"""Code adapted from International Brain Laboratory, T. (2021). ibllib [Computer software]. https://github.com/int-brain-lab/ibllib -""" +"""Code adapted from International Brain Laboratory, T. (2021). ibllib [Computer software]. https://github.com/int-brain-lab/ibllib""" import numpy as np diff --git a/notebooks/tutorial.ipynb b/notebooks/tutorial.ipynb index 19b7b615..09d41f44 100644 --- a/notebooks/tutorial.ipynb +++ b/notebooks/tutorial.ipynb @@ -9,52 +9,53 @@ "\n", "#### Open-source data pipeline for processing and analyzing extracellular electrophysiology datasets.\n", "\n", - "Welcome to the tutorial for the DataJoint Element for extracellular array electrophysiology. This\n", - "tutorial aims to provide a comprehensive understanding of the open-source data pipeline\n", - "created using `element-array-ephys`.\n", + "Welcome to the tutorial for the DataJoint Element for extracellular array\n", + "electrophysiology. This tutorial aims to provide a comprehensive understanding of the\n", + "open-source data pipeline created using `element-array-ephys`.\n", "\n", - "This package is designed to seamlessly process, ingest, and track extracellular electrophysiology\n", - "data, along with its associated probe and recording metadata. By the end of this\n", - "tutorial you will have a clear grasp on setting up and integrating `element-array-ephys`\n", - "into your specific research projects and lab. \n", + "This package is designed to seamlessly process, ingest, and track extracellular\n", + "electrophysiology data, along with its associated probe and recording metadata. By the\n", + "end of this tutorial you will have a clear grasp on setting up and integrating\n", + "`element-array-ephys` into your specific research projects and lab.\n", "\n", "![flowchart](../images/diagram_flowchart.svg)\n", "\n", "### Prerequisites\n", "\n", - "Please see the [datajoint tutorials GitHub\n", - "repository](https://github.com/datajoint/datajoint-tutorials/tree/main) before\n", - "proceeding.\n", + "Please see the\n", + "[datajoint tutorials GitHub repository](https://github.com/datajoint/datajoint-tutorials/tree/main)\n", + "before proceeding.\n", "\n", "A basic understanding of the following DataJoint concepts will be beneficial to your\n", - "understanding of this tutorial: \n", + "understanding of this tutorial:\n", + "\n", "1. The `Imported` and `Computed` tables types in `datajoint-python`.\n", - "2. The functionality of the `.populate()` method. \n", + "2. The functionality of the `.populate()` method.\n", "\n", "#### **Tutorial Overview**\n", "\n", - "+ Setup\n", - "+ *Activate* the DataJoint pipeline.\n", - "+ *Insert* subject, session, and probe metadata.\n", - "+ *Populate* electrophysiology recording metadata.\n", - "+ Run the clustering task.\n", - "+ Curate the results (optional).\n", - "+ Visualize the results.\n", + "- Setup\n", + "- _Activate_ the DataJoint pipeline.\n", + "- _Insert_ subject, session, and probe metadata.\n", + "- _Populate_ electrophysiology recording metadata.\n", + "- Run the clustering task.\n", + "- Curate the results (optional).\n", + "- Visualize the results.\n", "\n", "### **Setup**\n", "\n", "This tutorial examines extracellular electrophysiology data acquired with `OpenEphys`\n", - "and spike-sorted using Kilosort 2.5. The goal is to store, track\n", - "and manage sessions of array electrophysiology data, including spike sorting results and\n", - "unit-level visualizations. \n", + "and spike-sorted using Kilosort 2.5. The goal is to store, track and manage sessions of\n", + "array electrophysiology data, including spike sorting results and unit-level\n", + "visualizations.\n", "\n", - "The results of this Element can be combined with **other modalities** to create\n", - "a complete, customizable data pipeline for your specific lab or study. For instance, you\n", + "The results of this Element can be combined with **other modalities** to create a\n", + "complete, customizable data pipeline for your specific lab or study. For instance, you\n", "can combine `element-array-ephys` with `element-calcium-imaging` and\n", "`element-deeplabcut` to characterize the neural activity along with markless\n", "pose-estimation during behavior.\n", "\n", - "Let's start this tutorial by importing the packages necessary to run the notebook." + "Let's start this tutorial by importing the packages necessary to run the notebook.\n" ] }, { @@ -75,7 +76,7 @@ "source": [ "If the tutorial is run in Codespaces, a private, local database server is created and\n", "made available for you. This is where we will insert and store our processed results.\n", - "Let's connect to the database server." + "Let's connect to the database server.\n" ] }, { @@ -113,8 +114,8 @@ "### **Activate the DataJoint Pipeline**\n", "\n", "This tutorial activates the `ephys_acute.py` module from `element-array-ephys`, along\n", - "with upstream dependencies from `element-animal` and `element-session`. Please refer to the\n", - "[`tutorial_pipeline.py`](./tutorial_pipeline.py) for the source code." + "with upstream dependencies from `element-animal` and `element-session`. Please refer to\n", + "the [`tutorial_pipeline.py`](./tutorial_pipeline.py) for the source code.\n" ] }, { @@ -131,7 +132,7 @@ "metadata": {}, "source": [ "We can represent the tables in the `probe` and `ephys` schemas as well as some of the\n", - "upstream dependencies to `session` and `subject` schemas as a diagram." + "upstream dependencies to `session` and `subject` schemas as a diagram.\n" ] }, { @@ -590,16 +591,18 @@ "metadata": {}, "source": [ "As evident from the diagram, this data pipeline encompasses tables associated with\n", - "recording and probe metadata, results of clustering. A few tables, such as `subject.Subject` or `session.Session`,\n", - "while important for a complete pipeline, fall outside the scope of the `element-array-ephys`\n", - "tutorial, and will therefore, not be explored extensively here. The primary focus of\n", - "this tutorial will be on the `probe` and `ephys` schemas.\n", + "recording and probe metadata, results of clustering. A few tables, such as\n", + "`subject.Subject` or `session.Session`, while important for a complete pipeline, fall\n", + "outside the scope of the `element-array-ephys` tutorial, and will therefore, not be\n", + "explored extensively here. The primary focus of this tutorial will be on the `probe` and\n", + "`ephys` schemas.\n", "\n", "### **Insert subject, session, and probe metadata**\n", "\n", "Let's start with the first table in the schema diagram (i.e. `subject.Subject` table).\n", "\n", - "To know what data to insert into the table, we can view its dependencies and attributes using the `.describe()` and `.heading` methods." + "To know what data to insert into the table, we can view its dependencies and attributes\n", + "using the `.describe()` and `.heading` methods.\n" ] }, { @@ -757,9 +760,8 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "The cells above show all attributes of the subject table.\n", - "We will insert data into the\n", - "`subject.Subject` table. " + "The cells above show all attributes of the subject table. We will insert data into the\n", + "`subject.Subject` table.\n" ] }, { @@ -874,8 +876,8 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Let's repeat the steps above for the `Session` table and see how the output varies between\n", - "`.describe` and `.heading`." + "Let's repeat the steps above for the `Session` table and see how the output varies\n", + "between `.describe` and `.heading`.\n" ] }, { @@ -928,7 +930,9 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Notice that `describe`, displays the table's structure and highlights its dependencies, such as its reliance on the `Subject` table. These dependencies represent foreign key references, linking data across tables.\n", + "Notice that `describe`, displays the table's structure and highlights its dependencies,\n", + "such as its reliance on the `Subject` table. These dependencies represent foreign key\n", + "references, linking data across tables.\n", "\n", "On the other hand, `heading` provides an exhaustive list of the table's attributes. This\n", "list includes both the attributes declared in this table and any inherited from upstream\n", @@ -936,7 +940,8 @@ "\n", "With this understanding, let's move on to insert a session associated with our subject.\n", "\n", - "We will insert into the `session.Session` table by passing a dictionary to the `insert1` method." + "We will insert into the `session.Session` table by passing a dictionary to the `insert1`\n", + "method.\n" ] }, { @@ -1050,7 +1055,11 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Every experimental session produces a set of data files. The purpose of the `SessionDirectory` table is to locate these files. It references a directory path relative to a root directory, defined in `dj.config[\"custom\"]`. More information about `dj.config` is provided in the [documentation](https://datajoint.com/docs/elements/user-guide/)." + "Every experimental session produces a set of data files. The purpose of the\n", + "`SessionDirectory` table is to locate these files. It references a directory path\n", + "relative to a root directory, defined in `dj.config[\"custom\"]`. More information about\n", + "`dj.config` is provided in the\n", + "[documentation](https://datajoint.com/docs/elements/user-guide/).\n" ] }, { @@ -1158,10 +1167,9 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "As the Diagram indicates, the tables in the `probe` schemas need to\n", - "contain data before the tables in the `ephys` schema accept any data. Let's\n", - "start by inserting into `probe.Probe`, a table containing metadata about a\n", - "multielectrode probe. " + "As the Diagram indicates, the tables in the `probe` schemas need to contain data before\n", + "the tables in the `ephys` schema accept any data. Let's start by inserting into\n", + "`probe.Probe`, a table containing metadata about a multielectrode probe.\n" ] }, { @@ -1269,8 +1277,8 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "The probe metadata is used by the downstream `ProbeInsertion` table which we\n", - "insert data into in the cells below:" + "The probe metadata is used by the downstream `ProbeInsertion` table which we insert data\n", + "into in the cells below:\n" ] }, { @@ -1432,7 +1440,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Confirm the inserted data:" + "Confirm the inserted data:\n" ] }, { @@ -1543,7 +1551,8 @@ "### **Populate electrophysiology recording metadata**\n", "\n", "In the upcoming cells, the `.populate()` method will automatically extract and store the\n", - "recording metadata for each experimental session in the `ephys.EphysRecording` table and its part table `ephys.EphysRecording.EphysFile`." + "recording metadata for each experimental session in the `ephys.EphysRecording` table and\n", + "its part table `ephys.EphysRecording.EphysFile`.\n" ] }, { @@ -1776,7 +1785,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Let's view the information was entered into each of these tables:" + "Let's view the information was entered into each of these tables:\n" ] }, { @@ -2010,7 +2019,7 @@ "`ClusteringParamSet`. This table keeps track of all combinations of your spike sorting\n", "parameters. You can choose which parameters are used during processing in a later step.\n", "\n", - "Let's view the attributes and insert data into `ephys.ClusteringParamSet`." + "Let's view the attributes and insert data into `ephys.ClusteringParamSet`.\n" ] }, { @@ -2180,11 +2189,11 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "DataJoint uses a `ClusteringTask` table to\n", - "manage which `EphysRecording` and `ClusteringParamSet` should be used during processing. \n", + "DataJoint uses a `ClusteringTask` table to manage which `EphysRecording` and\n", + "`ClusteringParamSet` should be used during processing.\n", "\n", - "This table is important for defining several important aspects of\n", - "downstream processing. Let's view the attributes to get a better understanding. " + "This table is important for defining several important aspects of downstream processing.\n", + "Let's view the attributes to get a better understanding.\n" ] }, { @@ -2219,13 +2228,14 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "The `ClusteringTask` table contains two important attributes: \n", - "+ `paramset_idx` - Allows the user to choose the parameter set with which you want to\n", + "The `ClusteringTask` table contains two important attributes:\n", + "\n", + "- `paramset_idx` - Allows the user to choose the parameter set with which you want to\n", " run spike sorting.\n", - "+ `task_mode` - Can be set to `load` or `trigger`. When set to `load`, running the\n", + "- `task_mode` - Can be set to `load` or `trigger`. When set to `load`, running the\n", " Clustering step initiates a search for existing output files of the spike sorting\n", " algorithm defined in `ClusteringParamSet`. When set to `trigger`, the processing step\n", - " will run spike sorting on the raw data." + " will run spike sorting on the raw data.\n" ] }, { @@ -2249,7 +2259,8 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Let's call populate on the `Clustering` table which checks for kilosort results since `task_mode=load`." + "Let's call populate on the `Clustering` table which checks for kilosort results since\n", + "`task_mode=load`.\n" ] }, { @@ -2275,8 +2286,8 @@ "source": [ "### **Populate the results**\n", "\n", - "Once the `Clustering` table receives an entry, we can populate the remaining\n", - "tables in the workflow including `CuratedClustering`, `WaveformSet`, and `LFP`. " + "Once the `Clustering` table receives an entry, we can populate the remaining tables in\n", + "the workflow including `CuratedClustering`, `WaveformSet`, and `LFP`.\n" ] }, { @@ -2307,25 +2318,25 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Now that we've populated the tables in this DataJoint pipeline, there are one of\n", - "several next steps. If you have an existing pipeline for\n", - "aligning waveforms to behavior data or other stimuli, you can easily\n", - "invoke `element-event` or define your custom DataJoint tables to extend the\n", - "pipeline." + "Now that we've populated the tables in this DataJoint pipeline, there are one of several\n", + "next steps. If you have an existing pipeline for aligning waveforms to behavior data or\n", + "other stimuli, you can easily invoke `element-event` or define your custom DataJoint\n", + "tables to extend the pipeline.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "### **Visualize the results**" + "### **Visualize the results**\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "In this tutorial, we will do some exploratory analysis by fetching the data from the database and creating a few plots." + "In this tutorial, we will do some exploratory analysis by fetching the data from the\n", + "database and creating a few plots.\n" ] }, { @@ -2342,10 +2353,10 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "In the query above, we fetch a single `lfp_mean` attribute from the `LFP` table.\n", - "We also restrict the query to insertion number 1.\n", + "In the query above, we fetch a single `lfp_mean` attribute from the `LFP` table. We also\n", + "restrict the query to insertion number 1.\n", "\n", - "Let's go ahead and plot the LFP mean." + "Let's go ahead and plot the LFP mean.\n" ] }, { @@ -2377,15 +2388,16 @@ "metadata": {}, "source": [ "DataJoint queries are a highly flexible tool to manipulate and visualize your data.\n", - "After all, visualizing traces or generating rasters is likely just the start of\n", - "your analysis workflow. This can also make the queries seem more complex at\n", - "first. However, we'll walk through them slowly to simplify their content in this notebook. \n", + "After all, visualizing traces or generating rasters is likely just the start of your\n", + "analysis workflow. This can also make the queries seem more complex at first. However,\n", + "we'll walk through them slowly to simplify their content in this notebook.\n", "\n", "The examples below perform several operations using DataJoint queries:\n", + "\n", "- Fetch the primary key attributes of all units that are in `insertion_number=1`.\n", "- Use **multiple restrictions** to fetch timestamps and create a raster plot.\n", - "- Use a **join** operation and **multiple restrictions** to fetch a waveform\n", - " trace, along with unit data to create a single waveform plot" + "- Use a **join** operation and **multiple restrictions** to fetch a waveform trace,\n", + " along with unit data to create a single waveform plot\n" ] }, { @@ -2430,8 +2442,8 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Below we will use two queries to fetch *all* of the information about a single unit and\n", - "plot the unit waveform." + "Below we will use two queries to fetch _all_ of the information about a single unit and\n", + "plot the unit waveform.\n" ] }, { @@ -2694,33 +2706,32 @@ "source": [ "## Summary\n", "\n", - "Following this tutorial, we have: \n", - "+ Covered the essential functionality of `element-array-ephys`.\n", - "+ Learned how to manually insert data into tables.\n", - "+ Executed and ingested results of spike sorting with Kilosort.\n", - "+ Visualized the results. \n", + "Following this tutorial, we have:\n", + "\n", + "- Covered the essential functionality of `element-array-ephys`.\n", + "- Learned how to manually insert data into tables.\n", + "- Executed and ingested results of spike sorting with Kilosort.\n", + "- Visualized the results.\n", "\n", "#### Documentation and DataJoint Tutorials\n", "\n", - "+ [Detailed documentation on\n", - " `element-array-ephys`.](https://datajoint.com/docs/elements/element-array-ephys/)\n", - "+ [General `datajoint-python`\n", - " tutorials.](https://github.com/datajoint/datajoint-tutorials) covering fundamentals,\n", - " such as table tiers, query operations, fetch operations, automated computations with the\n", - " make function, and more.\n", - "+ [Documentation for\n", - " `datajoint-python`.](https://datajoint.com/docs/core/datajoint-python/)\n", + "- [Detailed documentation on `element-array-ephys`.](https://datajoint.com/docs/elements/element-array-ephys/)\n", + "- [General `datajoint-python` tutorials.](https://github.com/datajoint/datajoint-tutorials)\n", + " covering fundamentals, such as table tiers, query operations, fetch operations,\n", + " automated computations with the make function, and more.\n", + "- [Documentation for `datajoint-python`.](https://datajoint.com/docs/core/datajoint-python/)\n", "\n", "##### Run this tutorial on your own data\n", "\n", "To run this tutorial notebook on your own data, please use the following steps:\n", - "+ Download the [mysql-docker image for\n", - " DataJoint](https://github.com/datajoint/mysql-docker) and run the container according\n", - " to the instructions provide in the repository.\n", - "+ Create a fork of this repository to your GitHub account.\n", - "+ Clone the repository and open the files using your IDE.\n", - "+ Add a code cell immediately after the first code cell in the notebook - we will setup\n", - " the local connection using this cell. In this cell, type in the following code. \n", + "\n", + "- Download the\n", + " [mysql-docker image for DataJoint](https://github.com/datajoint/mysql-docker) and run\n", + " the container according to the instructions provide in the repository.\n", + "- Create a fork of this repository to your GitHub account.\n", + "- Clone the repository and open the files using your IDE.\n", + "- Add a code cell immediately after the first code cell in the notebook - we will setup\n", + " the local connection using this cell. In this cell, type in the following code.\n", "\n", "```python\n", "import datajoint as dj\n", @@ -2733,13 +2744,13 @@ "dj.conn()\n", "```\n", "\n", - "+ Run the code block above and proceed with the rest of the notebook." + "- Run the code block above and proceed with the rest of the notebook.\n" ] } ], "metadata": { "kernelspec": { - "display_name": "python3p10", + "display_name": "element-dlc", "language": "python", "name": "python3" }, @@ -2753,14 +2764,9 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.9.17" + "version": "3.9.18" }, - "orig_nbformat": 4, - "vscode": { - "interpreter": { - "hash": "ff52d424e56dd643d8b2ec122f40a2e279e94970100b4e6430cb9025a65ba4cf" - } - } + "orig_nbformat": 4 }, "nbformat": 4, "nbformat_minor": 2