Streaming Processing
When we develop these functionalities we are thinking about facilities. Currently, scipion is being used every day in several facilities in Europe, and more are joining from US and Canada. If you are running a Cryo EM facility and want more info, please Contact us. We will be happy to help you run Scipion there.
In the last months, we have been working on an extension of Scipion to process data in streaming, i.e, at the same time movies (or micrographs) are coming from the microscope PC. This allows to overlap computing time with the acquisition (reducing computational needs) and also to detect problems at early stages. This idea is implemented in different labs mainly by using custom-made scripts. The advantage of our Scipion solution is that you have the usual flexibility to choose what operations to do and the traceability to re-do some of the steps later. It is basically the same Scipion interface with one key change: the output is produced as soon as the first element is available, and it is later updated with new output elements. This allows concatenating several operations before the first one is completed.
On top of that, we have added the concept of monitors, the special protocols that constantly check how the execution of other protocols is going. We have developed several GUIs that are refreshed periodically and produce a graphical summary (e.g, CTF defocus values, system load etc). A summary is also generated in HTML format that can be easily copied to a public website to provide access for external users. (See an example here).
NOTE: This information is only available since today at branch: release-1.1.facilities-devel.
To run the demo as it is you need to have installed:
-
motioncor2
-
ctffind4
-
eman
-
jmbFalconMovies dataset
scipion install motioncor2 ctffind4 eman scipion testdata --download jmbFalconMovies
After installing all the requirements you just have to run:
scipion demo
This will pop up a small wizard like the one below ready to go.
You can leave all the displayed data untouched since it goes right with the test data. Once you click on the "Start demo" button scipion should appear with the new project loaded running in streaming mode:
Import is already importing files and the rest are scheduled. As soon as there is any input available, the protocols will start processing it and making it available for the next protocol in line.
The "Monitor summary" is monitoring the progress and generating an HTML report with the outcome of the data.