Timing information

Here we can include summaries or plots that shows how long it takes to process each step of the pipeline and see what can we make more efficient if possible.

Setting MS order with setToCasaOrder?

General conclusions:

Don't use setToCasaOrder. It takes a lot of time and in fact it does not make things faster. Even UVFIX immediately after loading the data take exactly the same ammount of time.
In general, currently the slowest step is aoflagger. It operates in write mode because there is not enough RAM.
After splitting the data to 128 chan/spw and 2s the calibration is really fast. About 40 minutes including plotting the calibration tables with plotcal.

Details:

The test was conducted on richards (12 CPU @ 3.50GHz, 64 GB of RAM). The data set if 184 GB in the original fits format.

Times to run different steps with setToCasaOrder: summary_test2_1

Times to run different steps with NO setToCasaOrder: summary_test2_3

Test conclusions:

setToCasaOrder takes a long time. About 3h spent on setting the order of a 184 GB file.
Most of the following steps with unaveraged data take the same ammount of time.
Aoflagger seems to take much longer on ordered data. Making it more than 1h slower.
The split step (including averaging to 128chan/spw and 2sec) is also slower when applying setOrder.
The rest of the steps after averaging does not make a difference.
I see the same behaviour on unaveraged data.