Break up long-running query in DR1 notebook #271
Conversation
|
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
|
@stephjuneau, @jacquesalice, this PR seems to have been forgotten. Could you please take a look? |
There was a problem hiding this comment.
@weaverba137 thanks for the reminder!
I ran this notebook on gp13. The long-running query did finish successfully for me, but it took 21 minutes to complete. And at the "Retrieving results" part of the async_ run, I got this message from Jupyter Lab:
IOPub data rate exceeded.
The Jupyter server will temporarily stop sending output
to the client in order to avoid crashing it.
To change this limit, set the config variable
`--ServerApp.iopub_data_rate_limit`.
Current values:
ServerApp.iopub_data_rate_limit=50000000.0 (bytes[/sec](https://gp13.datalab.noirlab.edu/sec))
ServerApp.rate_limit_window=5.0 (secs)
...Not sure if this is a red flag or if it's fine?
|
I've seen the warning message before, and it is just a warning. I think configuration of the Jupyter Lab server would need to be changed to get rid of it. |
|
Can I merge this? Or @stephjuneau did you still want to review? |
This PR fixes #270.
The compound query is split up into an inner query which can run asynchronously with the intermediate results stored in MyDB.
Although a solution involving
lastnightfromdesi_dr1.ztilewould be possible in principle, the plots resulting from this query expect MJD, and I didn't feel like doing a lot of extra math to convertlastnight(YYYYMMDD) into a day number like MJD.