Streaming Processing

This page is still under construction. We will add more information and some screenshots in a near future.

Processing in Streaming

In the last months we have been working on an extension of Scipion to process data in streaming, i.e, at the same time movies (or micrographs) are coming from the microscope PC. This allows to overlap computing time with acquisition (reducing computational needs) and also to detect problems at early stages. This idea is implemented in different labs mainly by using custom-made scripts. The advantage of our Scipion solution, is that you have the usual flexibility to choose what operations to do and the traceability to re-do some of the steps later. It is basically the same Scipion interface with one key change: the output is produced as soon as the first element is available, and it is later updated with new output elements. This allows to concatenate several operations before the first one is completed.

On top of that, we have added the concept of monitors, the special protocols that constantly check how the execution of other protocols is going. We have developed several GUIs that are refreshed periodically and produce a graphical summary (e.g, CTF defocus values, system load etc). A summary is also generated in HTML format that can be easily copied to a public web-site to provide access for external users. (See an example here).