You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository was archived by the owner on Feb 26, 2025. It is now read-only.
Please make sure to install the project requirements,
113
-
see the [dependencies](./README.md#dependencies) section in top README.
111
+
The connectome manipulation framework has been developed using reusable primitives, such as Python classes and specific file structures forindividual code modules that allow easy extension of the frameworkin order to add new functionality. Specifically, new types of (stochastic) models, tools for fitting them, new manipulation operations, and additional structural validation methods can be added to the code repository as outlined below:
114
112
115
-
This section applies to both Python versions 2 and 3.
113
+
## Models
116
114
117
-
## Setup
115
+
All models are implemented under [`/model_building/model_types.py`](connectome_manipulator/model_building/model_types.py) and are derived from an abstract base class `AbstractModel` which provides general functionality for loading/saving models and evaluating them, i.e., returning the model output given its input. Specific functionality must be implemented in a respective derived class which must define the model parameter names (`param_names`; i.e., variables storing the internal representation of the model), default parameter values (`param_defaults`), names of data frames (`data_names`; for large data elements, if any, that would be stored as associated HDF5 file), and input names (`input_names`; i.e., input variables the model output depends on). Moreover, the derived class must provide implementations of `get_model_output()` for returning the model output given its input variables, and `__str__()` for returning a string representation describing the model. When initializing a concrete model instance, values for all specified model parameters and data frames must be provided. Values for model parameters can be omitted in case default parameter values have been defined instead.
118
116
119
-
It is recommended to use `virtualenv`to develop ina sandbox environment:
117
+
Another useful (abstract) base class `PathwayModel` exists which can be used in the same way as outlined above, but which already includes pathway-specific model parameterization. Specifically, it allows to store different parameter values dependent on pairs of pre-synaptic (`src_type`) and post-synaptic (`tgt_type`) m-types, together with default values incase no pathway is specified.
120
118
121
-
```
122
-
virtualenv venv
123
-
. venv/bin/activate
124
-
pip install -r tests/requirement_tests.txt
125
-
```
119
+
## Model fitting functions
126
120
127
-
## Build
121
+
All model fitting functions are implemented as separate code modules (.py files) under [`/model_building`](connectome_manipulator/model_building) and must always contain the following functions for implementing the three steps of model building:
128
122
129
-
Run the following command to build incrementally the project: `pip install -e .`
123
+
- `extract()`forextracting relevant data (e.g., connection probabilities at binned distances) from a given connectome which will be stored automaticallyin a .pickle file by the framework
124
+
- `build()`for fitting model parameters against the data extracted during the previous step and initializing a model instance which will then be stored automatically as a .json file, optionally together with an associated HDF5 file
125
+
- `plot()`forgenerating visualizations of the extracted data versus the model output, and storing themin the output folder
130
126
131
-
## Test
127
+
Importantly, arbitrary parameters (optionally, including default values) can be added as keyword arguments to any of the three functions, values of which must be provided through a configuration file (see *Configuration file structure*in the [Documentation](https://connectome-manipulator.readthedocs.io/en/netneuro-24-0092-rev1/config_file_structure.html)) when launching model building.
132
128
133
-
Run the following command to run the Python unit-tests: `pytest tests`
129
+
## Manipulations
134
130
135
-
## Coding conventions
131
+
All manipulations are derived from an abstract base class `Manipulation` which is implemented in [`/connectome_manipulation/manipulation/base.py`](connectome_manipulator/connectome_manipulation/manipulation/base.py). The base class provides access to the neurons of a network model (through `self.nodes`) as well as to the input (i.e., before a manipulation) and output (i.e., after a manipulation) synapse tables (through `self.writer`). An alternative (abstract) base class, `MorphologyCachingManipulation`, exists which additionally provides efficient access to morphologies (through `self._get_tgt_morphs`) including a caching mechanism, i.e., without reloading them from the file system incase of repeated invocations.
136
132
137
-
The code coverage of the Python unit-tests may not decrease over time.
138
-
It means that every change must go with their corresponding Python unit-tests to
139
-
validate the library behavior as well as to demonstrate the API usage.
133
+
A concrete manipulation must be implemented in a derived classes and stored in a separate code module (.py file) under [`/connectome_manipulation/manipulation`](connectome_manipulator/connectome_manipulation/manipulation). It must contain an implementation forthe `apply()` method which must return a new synapse table (through `self.writer`) as a result of the manipulation. Importantly, arbitrary parameters (optionally, including default values) can be added as keyword arguments to the `apply()` method, values of which must be provided through a configuration file (see *Configuration file structure*in the [Documentation](https://connectome-manipulator.readthedocs.io/en/netneuro-24-0092-rev1/config_file_structure.html)) when launching a manipulation.
134
+
135
+
## Structural comparison functions
136
+
137
+
All structural comparison functions are implemented as separate code modules (.py files) under [`/connectome_comparison`](connectome_manipulator/connectome_comparison) and must always contain functions for implementing the two following steps:
138
+
139
+
- `compute()`for computing specific structural features from a given connectome (e.g., connection probability by layer), which will be evaluated for both connectomes to compare and results of which will be automatically stored as .pickle files by the framework
140
+
- `plot()`forplotting a graphical representation of individual feature instances (e.g., 2D matrix plot of connection probabilities by layer) or the difference between two such instances, which will be automatically storedin a compound output figure when comparing two connectomes
141
+
142
+
Importantly, arbitrary parameters (optionally, including default values) can be added as keyword arguments to the two functions, values of which must be provided through a configuration file (see *Configuration file structure*in the [Documentation](https://connectome-manipulator.readthedocs.io/en/netneuro-24-0092-rev1/config_file_structure.html)) when launching a structural comparison.
Copy file name to clipboardExpand all lines: README.rst
+52-15Lines changed: 52 additions & 15 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -24,14 +24,15 @@ Table of contents
24
24
- `Structural comparison`_
25
25
26
26
5. `Examples`_
27
-
6. `How to contribute`_
28
-
7. `Citation`_
29
-
8. `Publications that use or mention Connectome-Manipulator`_
27
+
6. `Documentation`_
28
+
7. `How to contribute`_
29
+
8. `Citation`_
30
+
9. `Publications that use or mention Connectome-Manipulator`_
30
31
31
32
- `Scientific papers that use Connectome-Manipulator`_
32
33
- `Posters that use Connectome-Manipulator`_
33
34
34
-
9. `Funding & Acknowledgment`_
35
+
10. `Funding & Acknowledgment`_
35
36
36
37
Introduction
37
38
------------
@@ -62,6 +63,11 @@ All dependencies declared in ``setup.py`` and are available from PyPI, including
62
63
63
64
Recommended Python version: v3.10.8
64
65
66
+
❗ Compatibility notes
67
+
~~~~~~~~~~~~~~~~~~~~~~
68
+
69
+
The software famework is intended to be used on Linux/MacOS-based systems! Specifically, some dependencies, like ``libsonata``, are currently not compatible with Microsoft Windows OS.
70
+
65
71
Framework overview
66
72
------------------
67
73
@@ -73,11 +79,11 @@ consists of the following main components:
73
79
74
80
- | **Connectome manipulator**
75
81
|As specified in the config, applies one or a sequence of manipulations to a given SONATA connectome, and writes the manipulated connectome to a new SONATA edges file. All manipulations are separately implemented in sub-modules and can be easily extended.
76
-
|Details can be found in the corresponding README file in the repository: `connectome_manipulation/README.md <https://github.com/BlueBrain/connectome-manipulator/blob/main/connectome_manipulator/connectome_manipulation/README.md>`_
82
+
|Details can be found in the corresponding README file in the repository: `connectome_manipulation/README.md <connectome_manipulator/connectome_manipulation/README.md>`_
77
83
78
84
- | **Model building**
79
85
|As specified in the config, builds a model from a given connectome and writes the model to a file to be loaded and used by specific manipulations requiring a model (e.g., model-based rewiring based on connection probability model). All models are separately implemented in sub-modules and can be easily extended.
80
-
|Details can be found in the corresponding README file in the repository: `model_building/README.md <https://github.com/BlueBrain/connectome-manipulator/blob/main/connectome_manipulator/model_building/README.md>`_
86
+
|Details can be found in the corresponding README file in the repository: `model_building/README.md <connectome_manipulator/model_building/README.md>`_
81
87
82
88
Notes:
83
89
@@ -87,7 +93,9 @@ consists of the following main components:
87
93
88
94
- | **Structural comparator**
89
95
|As specified in the config, performs a structural comparison of the original and manipulated connectomes. Different structural parameters to compare (connection probability, synapses per connection, ...) are separately implemented in sub-modules and can be easily extended.
90
-
|Details can be found in the corresponding README file in the repository: `connectome_comparison/README.md <https://github.com/BlueBrain/connectome-manipulator/blob/main/connectome_manipulator/connectome_comparison/README.md>`_
96
+
|Details can be found in the corresponding README file in the repository: `connectome_comparison/README.md <connectome_manipulator/connectome_comparison/README.md>`_
97
+
98
+
The structure of the respective configuration files can be found under `doc/source/config_file_structure.rst <doc/source/config_file_structure.rst>`_
91
99
92
100
ℹ️ More details can be also found in the accompanying publication (esp.
93
101
*Supplementary tables*), see `Citation`_.
@@ -180,6 +188,24 @@ Please note that this feature will require at least 4 MPI ranks. Dask will use 2
180
188
181
189
When processing with ``parallel-manipulator``, one may pass the flag ``--target-payload`` to determine how big the individual workload for each process should be. The default value of 20e9 was determined empirically to run on the whole mouse brain with 75 million neurons. We recommend to use this value as a starting point and scale it up or down to achieve the desired runtime characteristics.
182
190
191
+
Details on the CONFIG file structure can be found under `doc/source/config_file_structure.rst <doc/source/config_file_structure.rst>`_
192
+
193
+
❗ Notes on error handling
194
+
^^^^^^^^^^^^^^^^^^^^^^^^^^
195
+
196
+
Errors may occur for many different reasons and are not always easy to track. Most common errors are that an allocation gets "killed", either due to a time limit or due to an out-of-memory error. Here we provide a few hints on how to avoid or track errors that may occur:
197
+
198
+
- Use the "verbose" mode (``-v`` flag) which will produce a lot of log output.
199
+
- Look into the log files: there is usually one master log file and individual log files for all data splits, all of which can be found in the ``/logs`` subfolder of the output circuit folder.
200
+
- Use a small connectome to start with.
201
+
- Use a simple operation to start with, such as ``null_manipulation`` (see examples).
202
+
- Run serially to start with, before switching to parallel processing.
203
+
- Start with a single data split.
204
+
- But: In case of memory errors, use more than a single data splits, even when running serially (!), which will reduce the memory consumption as individual splits will be processed one after the other.
205
+
- When running in parallel, use ``--tasks-per-node`` in the SLURM configuration to define how many tasks (=splits) will be executed on a single node; reducing this number may reduce the risk of out-of-memory errors.
206
+
- In general: Increasing memory allocation and/or allocation time may help.
207
+
- For high performance: Allocate many nodes and use parallel processing together with a relatively large number of data splits depending on the network size (i.e., aim for a few hundered post-synaptic neurons per data split).
208
+
183
209
Model building
184
210
~~~~~~~~~~~~~~
185
211
@@ -190,9 +216,13 @@ Model building
190
216
Extract and build models from existing connectomes.
191
217
192
218
Options:
193
-
--force-reextract Force re-extraction of data, in case already existing.
194
-
--force-rebuild Force model re-building, in case already existing.
195
-
--help Show this message and exit.
219
+
--force-reextract Force re-extraction of data, in case already existing.
220
+
--force-rebuild Force model re-building, in case already existing.
221
+
--cv-folds INTEGER Optional number of cross-validation folds, overwrites
222
+
value in config file
223
+
--help Show this message and exit.
224
+
225
+
Details on the CONFIG file structure can be found under `doc/source/config_file_structure.rst <doc/source/config_file_structure.rst>`_
196
226
197
227
Structural comparison
198
228
~~~~~~~~~~~~~~~~~~~~~
@@ -210,15 +240,22 @@ Structural comparison
210
240
in case already existing.
211
241
--help Show this message and exit.
212
242
243
+
Details on the CONFIG file structure can be found under `doc/source/config_file_structure.rst <doc/source/config_file_structure.rst>`_
244
+
213
245
Examples
214
246
--------
215
247
216
-
Examples can be found under `examples/ <https://github.com/BlueBrain/connectome-manipulator/tree/main/examples>`_ in the repository.
248
+
Examples can be found under `examples/ </examples>`_ in the repository.
249
+
250
+
Documentation
251
+
-------------
252
+
253
+
The full documentation (API reference, CONFIG file structure, ...) can be found on `Read the Docs <https://connectome-manipulator.readthedocs.io/en/netneuro-24-0092-rev1>`_.
217
254
218
255
How to contribute
219
256
-----------------
220
257
221
-
Contribution guidelines can be found in `CONTRIBUTING.md <https://github.com/BlueBrain/connectome-manipulator/blob/main/CONTRIBUTING.md>`_ in the repository.
258
+
Contribution guidelines can be found in `CONTRIBUTING.md <CONTRIBUTING.md>`_ in the repository.
222
259
223
260
Citation
224
261
--------
@@ -244,11 +281,11 @@ Publications that use or mention Connectome-Manipulator
244
281
Scientific papers that use Connectome-Manipulator
245
282
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
246
283
247
-
- Michael W. Reimann, Sirio Bolaños-Puchet, Jean-Denis Courcol, Daniela Egas Santander, et al. (2022) **Modeling and Simulation of Neocortical Micro- and Mesocircuitry. Part I: Anatomy.** bioRxiv 2022.08.11.503144. DOI: `10.1101/2022.08.11.503144<https://doi.org/10.1101/2022.08.11.503144>`_
284
+
- Michael W. Reimann, Sirio Bolaños-Puchet, Jean-Denis Courcol, Daniela Egas Santander, et al. (2024) **Modeling and Simulation of Neocortical Micro- and Mesocircuitry. Part I: Anatomy.** eLife, 13:RP99688. DOI: `10.7554/eLife.99688.1<https://doi.org/10.7554/eLife.99688.1>`_
248
285
249
-
- James B. Isbister, András Ecker, Christoph Pokorny, Sirio Bolaños-Puchet, Daniela Egas Santander, et al. (2023) **Modeling and Simulation of Neocortical Micro- and Mesocircuitry.** Part II: Physiology and Experimentation. bioRxiv 2023.05.17.541168. DOI: `10.1101/2023.05.17.541168 <https://doi.org/10.1101/2023.05.17.541168>`_
286
+
- James B. Isbister, András Ecker, Christoph Pokorny, Sirio Bolaños-Puchet, Daniela Egas Santander, et al. (2023) **Modeling and Simulation of Neocortical Micro- and Mesocircuitry. Part II: Physiology and Experimentation.** bioRxiv, 2023.05.17.541168. DOI: `10.1101/2023.05.17.541168 <https://doi.org/10.1101/2023.05.17.541168>`_
250
287
251
-
- Daniela Egas Santander, Christoph Pokorny, András Ecker, Jānis Lazovskis, Matteo Santoro, Jason P. Smith, Kathryn Hess, Ran Levi, and Michael W. Reimann. (2024) **Efficiency and reliability in biological neural network architectures.** bioRxiv 2024.03.15.585196. DOI: `10.1101/2024.03.15.585196 <https://doi.org/10.1101/2024.03.15.585196>`_
288
+
- Daniela Egas Santander, Christoph Pokorny, András Ecker, Jānis Lazovskis, Matteo Santoro, Jason P. Smith, Kathryn Hess, Ran Levi, and Michael W. Reimann. (2024) **Efficiency and reliability in biological neural network architectures.** bioRxiv, 2024.03.15.585196. DOI: `10.1101/2024.03.15.585196 <https://doi.org/10.1101/2024.03.15.585196>`_
0 commit comments