Updates for May 23

[bfhealy]

Announcements

• A DR16 vs. DR5 comparison shows that ZTF_sources_20230309 on melman contains more total sources in more fields than ZTF_sources_20210401 on gloria. This is promising given the previous ingestion issues with the former catalog.
• DR16 features are being generated for the 20 SCoPe fields on Expanse. Currently, fields 296, 297, 434 and 424 are complete.

Pull requests

• Fix json.dump bug in catalog completeness code #380
• Add minobs filter to catalog completeness code #381
• Set default minimum period to 30 mins #382
• Enable inferencing on generated features #384

Issues

• Test currently failing due to melman catalog maintenance #335
• Modify inference code to run on generated features #383

Going forward

We will work on:

• Updating inference.py to run on generated features while refactoring the code to be more efficient
• Ingesting DR16 features into a new Kowalski collection
• Adapting train_xgb_slurm.py and train_xgb_job_submission.py for DNN with the goal of tuning hyperparameters on Expanse