Deployment Pipeline

[pgulley]

To start treating this like a real production product, let's build out some deployment scripts.
I'd like this to have the same basic interface as the deployment scripts @philbudne built out for web_search- a deploy script which will manage a release of a checked out version of the sous-chef package in dev, staging, and production realms.

I'd like to keep a single production prefect server, with production worker instances, but staging and various dev instances should spin up their own self contained prefect stack.

for Staging / Dev:

1. Deploy a Prefect server
2. Create a prefect profile locally to point to the prefect server instance
3. Deploy a prefect worker, named appropriately for the realm
4. Use the scripts/setup_secrets.py package to install the sous-chef secret values into the prefect server.
5. Deploy the sous-chef instance via prefect deploy- after interpolating some templated values reflecting the above steps into prefect.yaml

@philbudne, having a script that manages steps 1-3 would be a great start, and we can go from there?

[pgulley]

The one kink in this, when thinking about how to get the sous-chef kitchen going as well, is that that will be deployed from an entierly different repository, and with a different prefect.yaml being interpolated as a baseline. This package sort of serves double-duty as a standalone deployment and as a package that is invoked by other standalone deployments.

[philbudne]

I think the first thing to figure out is if either Dokku or docker-compose would work: Dokku: avoids Dockerfile for (single) image (but that image can be deployed multiple times running different startup commands) requires some fiddling for inter-container comms, multiple images postgresql available as turnkey service single server docker-compose: needs Dockerfile(s) inter-container name resolution, multiple images out of the box allows multi-server deployments

[philbudne]

re: sous-chef kitchen Several thoughts: 1. SCK has it's own Dockerfile This is probably my least favorite, as it means an independant moving part for versioning. 2. Reference/install the SCK repository in Python requirements/pyproject This is what we do with mc-providers and mc-manage. version control is explicit, and baked into the requirements file. 3. Use a git submodule (a recursive checkout of the base module brings in submodules, I think version control is explicit (the submodule reference includes the hash/tag/branch)?) We don't use it elsewhere, I have no experience with it, and I'd rather not introduce something new.

[pgulley]

Sounds like Dockerfiles are the way to go- we've had plenty of success with that model in the past.

Re:SCK, I unfortunately think that 1 is the closest to what I was thinking of. 2 and 3 wouldn't really work, as SCK has SC as a dependency- the core SC deployment process is just the SCK deployment process without the additional step of deploying the API- would it be possible to build the deploy scripts in a way where they could be invoked by a dependent supermodule, top reduce complexity?

ie:
Sous-Chef

• bin
  • deploy.sh (does the above steps)

Sous-Chef-Kitchen

• Sous-Chef (package install)
• bin
  • deploy.sh (reuses some of sc/bin/deploy.sh, with sck/prefect.yaml and additionally deploys the SCK APi)

Or perhaps there's just a prefect-deploy repository that holds all of the re-used logic, that's invoked by both sc and sck?

[philbudne]

I can't avoid thinking that SCK is a separate repo only by the artifact of the development process. I (STRONLY) think keeping them separate only adds pain and complexity when it comes to version alignment. If a developer doesn't want to run the API (avoiding installing many packages in the image build), then for developers there could be a deploy.sh --api=true/false option (short name -A) and the default could be taken from the user environment so that developers could override the built-in default.

[pgulley]

Well, there are two distinct use-cases being served by the two repositories- Right now SC is being used for one-off research questions, where the pipelines need to be spun up fast, iterated on, and then only run a very small number of times (hence loading the recipe from S3 or elsewhere down the line, so that a new recipe can be run without the need to perform a whole redeploy). The SCK case assumes a small, slowly growing number of recipes which are built into the deployment and can be invoked interactively by external services and users.
It could be that the functionality related to the research-supporting case should also belong in the SCK repository- that would probably simplify a bunch of this conversation. Maybe call that first case the "Caterer"- it's a very small amount of code that actually defines the deployment, so having it just as a module next to the Kitchen would be fine.

[philbudne]

The fewer the number of moving pieces (git repos, docker images) the easier the job will be (less checking and verifying). Regarding spinning up fast: if the source repo hasn't changed, the image shouldn't need to be rebuilt. If it doesn't make sense to have a single image, could it make sense for the SCK image to use the plain SC image as the base image?

[philbudne]

Is the work for this issue all done (in s-c-k)?