Automate Production Steps of LNSwap Resources

[AshtonStephens]

Overview

This ticket with track automating the production steps of LNSwap.

LNSwap

The current production infrastructure runs on Google Cloud Platform under the project lnswap and project number 517511623071. We've set the infrastructure up manually but are lacking some core production infrastructure to detect issues early and keep rolling updates of the stacks nodes.

1. We want to have our stacks nodes deployed and managed by some form of template
2. We want an early warning system to detect when our node is out of sync.
   1. Warn us when the head of our stacks node is behind the stacks foundation head
   2. Detect host errors

Additional requirements and steps will be added in this ticket.

[AshtonStephens]

I won't be able to address this ticket for a while while prioritizing the Nakamoto work. If possible, I'd assign this to someone who can look at it sooner. For someone unfamiliar with cloud infrastructure I'd recommend looking into the following:

1. Defining infrastructure as code for the LNSwap resouces. If we choose to go the generic route use TerraformCDK, and if we go the AWS route use AWS CDK. For both, prioritize using TypeScript because those are the primary focus of the CDK development teams and have the most tooling and support.
2. Look into making dashboard options for metrics coming from the cloud providers. That's a big painpoint for the current infrastructure. The biggest metrics (ask @pseudozach for details, this terminology is probably wrong):
   1. Successfully completed transactions
   2. LNSwap stacks head alignment with Hiro stacks head.
3. Approximate the load on the host computers and record it. Then allocate cloud infrastructure that matches that required load and can autoscale to fit an expected number of calls.

There are more steps but these are the immediate ones. In the short term I'd recommend cutting a ticket to make demo infrastructure and then using that for the LNSwap production environments.

The POC for LNSwap is @pseudozach, who currently has access to all the google cloud infrastructure for LNSwap. I (@AshtonStephens) have the most experience with cloud infrastructure and will be able to provide intermittent support.

newserversetup.md is a markdown that @pseudozach uses for spinning up the LNSwap hosts. It's important to use the LNSwap github repo instead of psuedozach github repo specified in the scripts, so the scripts aren't quite correct.

[AshtonStephens]

Assigning to @pseudozach as this falls under LNSwap and I won't be able to get to this for a while.