initiate monitoring early/now with diagnostic page

[zackarno]

@caldwellst @hannahker

I was thinking we might be able to remove some of the code complexity, by scratching the whole first_run component. I mean we can leave it in, but stop thinking about it if we just start deployment without any email components (for now).

I was thinking that I could just write a notebook that just gets auto published somewhere (like rpubs) that gives us an overview of run diagnostics. Could be like a global overview of the runs, datasets output, alerts... nothing fancy, but would just help us monitor how smoothly the backend is being implemented?

[caldwellst]

Not sure what you mean about starting deployment without any email components, but yeah, we are moving people onto the Mailchimp subscription and going live now basically. Don't think it makes much sense to talk about the system without email components particularly as determining recent alerts that we want to include in the system is based off of an understanding of what has already been sent out recently.

Yeah, first_run complexity is quite annoying, but now that is fully operational, we are able to have a synthetic historic dataset of signals, which we want to launch with, so don't see it being removed now. However, always interested in ways to reduce the code complexity if we have ideas on how to reduce the scope of it. However, probably best to wait til we have some decent testing infrastructure up and running before any major changes to ensure no unexpected shifts come from changing the codebase.

[zackarno]

but yeah, we are moving people onto the Mailchimp subscription and going live now basically.

oh fantastic. I want to sign up.

Yeah, first_run complexity is quite annoying, but now that is fully operational, we are able to have a synthetic historic dataset of signals, which we want to launch with, so don't see it being removed now.

ok i need to review the code a bit before responding to this one

particularly as determining recent alerts that we want to include in the system is based off of an understanding of what has already been sent out recently

that's right, i forgot we had switched to using the mailchimp api itself to determine what had been sent out rather than using flagging logic.. hmmm.

[zackarno]

that's right, i forgot we had switched to using the mailchimp api itself to determine what had been sent out rather than using flagging logic.. hmmm.

not mission critical or urgent, but it might make sense for us to reconsider this decision and essentially just create logging file that is in sync w/ mailchimp. This hard dependency on mailchimp servers might create some future limitations and added complexity for future applications data investigations

[caldwellst]

So, there is a flagging logic for each indicator, but also alerts are checked to make sure an alert at that same level has not been generated in the past 180 days and that it is recent (within the past 90 days). We aren't depending at all on Mailchimp servers, simply checking against the overarching signals.parquet file that is the record of all of our alerts. You can see this in the filter_alerts.R script.

[hannahker]

@zackarno @caldwellst what do to think of an additional status field for each signal? If we're treating the root level signals.parquet as our source of truth, it feels like we might be missing that information about what has actually happened with each signal (or at least it isn't intuitive to me yet from looking at the existing fields). The potential values in this field could be something like archive, draft, alerted, not alerted?

[caldwellst]

That could make sense! Currently, the status can be imputed, but having a field could be nice. In the root level signals.parquet, you will only have archive and alerted though. Currently, you differentiate by seeing if there is any email campaign (e.g. campaign_email_url and campaign_email_id is non-missing). draft is everything in the indicator level folders, they can never be alerted and are always in draft mode.

not alerted does not exist, the dataset is only for alerts that:

1. were generated in the first_run and tagged as archive
2. were actually sent out as alerts, so would be tagged as alerted

If during triaging of an alert, we decide not to send something out, then we would need to change that indicator's method of alerting so no signal is generated in the run (hence no campaigns or other data, and it's never a row in the output dataset).

[caldwellst]

And the idea of run diagnostics is nice. Not really sure exactly what you mean to show, but interested to hear more! Although hoping most of this is automatic by using Azure Databricks or Airflow or whatever we decide for our orchestration!

[caldwellst]

Erika is building a global map of the most recent alerts (past 3 months) that will sit on the Signals landing page, which will let us know where. But no alerts are being added without us manually curating them in the triage_signals.R script. That is, if an alert is created, it is only a draft, and it is only sent and added to the core dataset if and when we approve it going out.

Could still have the diagnostics of when and where we've alerted, but the map might cover it. I did some similar analysis in this slide deck to justify our selection procudures. Might be nice to have more like what you're saying to monitor, but probably some other things more critical in the outstanding issues, especially thinking through the testing!

[hannahker]

Overall run diagnostics/logging would definitely be helpful! I'm thinking about perhaps two sets of requirements here:

1. Traceability and overview for the alerts themselves. ie. When/where/why the system alerted. Seems like this could be more public facing, like the map that you mentioned?
2. Detailed logging throughout each run of the monitoring scripts for technical review and debugging. Assuming this would be injected at the R level, then collected and managed by whatever orchestration tool we're using?

We might also want to have some standardized logs on the triage_signals.R process. ie. Who approved the alert? When did this happen?

[zackarno]

Those 2 tracks make sense to me. If we make something additional to the map Erika is building I don't think it has to be that public facing per se at this stage, but rather a useful tool for us.

[caldwellst]

Yeah, nice tracks. On 1, the map is definitely designed to help users explore through the alerts data to exactly understand where and when we alerted. I think the why should be clear public documentation on methods for different indicators, and the design of the alerts themselves is intended to make it clear why we alerted. For instance, now in the structure rather than just a simple titling plots with what they describe, they have the key message we want to get across. And all of this will be embedded in the map that Erika creates, you can see the latest draft in the channel.

On 2, exactly! By default, we seem to get that in GitHub Actions and I assume we would get in whatever orchestration tool automatically? Or do you think we need to do specific things now or after we adopt it to improve?

I like the logs idea on triaging. I think just two new columns will suffice in the alerts data. triager, triage_time.

[caldwellst]

I have added in triage_approver and triage_time in the system now.