What is our procedure for triaging signals?

[hannahker]

What happens after a signal is detected? What's our procedure for sending out the Mailchimp campaign? We should have this clearly defined and documented.

So far in development, we've run the triage_signals() function locally to either APPROVE or DELETE a campaign. Approving will send out the campaign.

Some of my thoughts/questions:

• I don't love any dependence on manually running code locally. It's not traceable, is more error prone, we may have to wrestle with different local environments, and really limits the ownership of this process to technical team members. I don't have a good alternative in mind yet, however.
• Outside of the Data Science team, who needs to be involved and notified of this process within the Centre?

[caldwellst]

I don't love any dependence on manually running code locally. It's not traceable, is more error prone, we may have to wrestle with different local environments, and really limits the ownership of this process to technical team members. I don't have a good alternative in mind yet, however.

Certainly is error prone. We hope that {renv} reduces the differences across machines and ensures common infrastructure, but that isn't completely gone. An option could be that the process is instead run as a parameterised GitHub action based on indicator_id. In this instance, it wouldn't be triaging, it would be simply to approve and send or delete. I don't see an issue there at all, we would just want some convenience for opening up the campaign. Which is probably covered in #120 anyway if we have a link to the campaign generated in our notification!

really limits the ownership of this process to technical team members

On this point in particular, I think we should be clear that we as the data science team are owners of this process, however. And while in the above example anyone can click "Approve", if anyone can click "Delete", there is nothing non-technical people can do after that point. All fixes to regenerate the alert are technical in nature. So while people may be involved in reviewing the campaigns themselves for a time, I think it's risky to have anyone non-technical involved in this process and we should explicitly own this, because otherwise there is a risk of deletion occurring without signals being regenerated or changes being done, the critical follow-on procedure.

And I don't think we can get away from having scripted deletion in this way, because the files in Mailchimp are spread across different content, template, and campaign folders, and manually trying to edit them would be extremely risky. The other thing I"m thinking, the approval requiring manual typing of APPROVE or DELETE scripted is a nice check. If we went to GitHub actions or some other online process, we would need this type of setup. I imagine it's possible, but we couldn't have the process just be parameterised, because you could easily accidentally select either option and remove or send a campaign very easily.

Outside of the Data Science team, who needs to be involved and notified of this process within the Centre?

We are involving the wider Centre in approving the general content of the emails, but we will narrow this list of approvers for release.

Proposal

As above, I would propose that they are just involved in commenting on the draft campaign link, and we remain solely responsible for actually approving or deleting based on comments received. Deletion happens if any changes need to occur. The reviewers initially would consist of:

1. Ourselves
2. Leonardo
3. Erika
4. Sarah
5. Sharing to the #proj-hdx-signals Slack channel

Eventually, we will likely reduce triaging, starting from 5, and eventually reducing just to 1 when we have more trust in the system. This list could be used when implementing new indicators. I don't think we need to have a set timeline for when we start reducing reviewers, but something we will likely know when we feel comfortable.

[hannahker]

And while in the above example anyone can click "Approve", if anyone can click "Delete", there is nothing non-technical people can do after that point. All fixes to regenerate the alert are technical in nature. So while people may be involved in reviewing the campaigns themselves for a time, I think it's risky to have anyone non-technical involved in this process and we should explicitly own this, because otherwise there is a risk of deletion occurring without signals being regenerated or changes being done, the critical follow-on procedure.

Yes, this is a great point. I wasn't really thinking about the delete process as much...

This process sounds good to me to start off with. I don't think we need to over-engineer or over think things too much at this stage. As we've discussed previously, I think one of the most important things is to have that clear log of the lifecycle of each alert -- ie. if it's deleted or approved, when those actions happen, and by who, etc.

The signals dataset is a good place, but maybe eventually we can extend the Slack functionality to have more updates on what's going on. I think we could even put Approval buttons in messages... 🤔 Although we'd have to be very careful with permissions there.