Improved logging

[NicholasBratvold]

Made a few logging changes:

• A new log file is made for every new experiment. Previously a log file was created every time oracle was started.
• Removed print statements and replaced with appropriate log level.
• Updated log severity level of some logs to a more appropriate level.
• Added in periodic img_metadata logging. Keys and Period defined as a scope constant.
• Added experiment meta_data logging.
• Added end of run message with cell counts.
• Added some additional logs as needed.

[i-jey]

Wow - thanks for going through all this. Painstaking but long-needed. It looks good to me.

• One thing to add to Grafana and which requires no changes to this PR
• We currently output the net FPS of an experiment self.logger.info(f"Net FPS is {self.frame_count/runtime:.1f}").
• A sudden change in net FPS could alert us to other issues (say due to a performance regression in a recent PR, or some other issue with the scopes - maybe running too hot and building up queues, slow SSD, etc.)

Aside, the one comment I made on ewma_filtering_utils.py is unrelated to the contents of this PR, I'll submit a separate one to fix that minor bug.

[i-jey]

Setup script looks good!

[paul-lebel]

NBD but definitely more confusing that necessary. Could be:
if sample_type not in (CLINICAL_SAMPLE, CULTURED_SAMPLE):

[NicholasBratvold]

I agre, I will change that in another pull request but I want to keep this specific to logging.

[paul-lebel]

Is there a reason you converted some of these instances to logger.error() and some to logger.exception()? AFAIK the only difference is whether the stack is included by default or not.

[NicholasBratvold]

I converted all exceptions to errors so I believe there should be no logger.exceptions left. Previously, the severity levels were inconsistent so I tried to simplify things and keep more consistency in our severity level choices. Since we are also receiving alerts for tracebacks, I want them to be serious and not send us notifications for exceptions like the syringe being at its end of travel which tends to happen every run.

[paul-lebel]

ah ok. I thought I had seen one the other way around but I was wrong!

[paul-lebel]

Aside from minor comments above, looks good to me!

[NicholasBratvold]

Thanks for the review! I will merge it in.