From 4ab3d837f45096ef2ba45e31ada754c78cce7e7f Mon Sep 17 00:00:00 2001 From: nicktrn <55853254+nicktrn@users.noreply.github.com> Date: Tue, 24 Sep 2024 11:48:28 +0100 Subject: [PATCH 1/8] add docker only note --- docs/open-source-self-hosting.mdx | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/open-source-self-hosting.mdx b/docs/open-source-self-hosting.mdx index 44f3bea51b..59a3c8647a 100644 --- a/docs/open-source-self-hosting.mdx +++ b/docs/open-source-self-hosting.mdx @@ -3,6 +3,8 @@ title: "Self-hosting" description: "You can self-host Trigger.dev on your own infrastructure." --- +This guide is for Docker only. We don't currently provide documentation for Kubernetes. + ## Overview From 86f3b369a12bc4d458f0275912f15ccedd32f764 Mon Sep 17 00:00:00 2001 From: nicktrn <55853254+nicktrn@users.noreply.github.com> Date: Tue, 24 Sep 2024 11:48:43 +0100 Subject: [PATCH 2/8] add support section --- docs/open-source-self-hosting.mdx | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/open-source-self-hosting.mdx b/docs/open-source-self-hosting.mdx index 59a3c8647a..881d6b4772 100644 --- a/docs/open-source-self-hosting.mdx +++ b/docs/open-source-self-hosting.mdx @@ -15,6 +15,10 @@ The self-hosting guide comes in two parts. The first part is a simple setup wher You're going to need at least one Debian (or derivative) machine with Docker and Docker Compose installed. We'll also use Ngrok to expose the webapp to the internet. +## Support + +It's dangerous to go alone! Join the self-hosting channel on our [Discord server](https://discord.gg/NQTxt5NA7s). + ## Caveats The v3 worker components don't have ARM support yet. From 8ea634b7f08652c22ccd478b3db6867347f0b966 Mon Sep 17 00:00:00 2001 From: nicktrn <55853254+nicktrn@users.noreply.github.com> Date: Tue, 24 Sep 2024 12:08:40 +0100 Subject: [PATCH 3/8] add large payloads section --- docs/open-source-self-hosting.mdx | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/docs/open-source-self-hosting.mdx b/docs/open-source-self-hosting.mdx index 881d6b4772..a06f436bfc 100644 --- a/docs/open-source-self-hosting.mdx +++ b/docs/open-source-self-hosting.mdx @@ -289,6 +289,25 @@ echo "FORCE_CHECKPOINT_SIMULATION=0" >> .env ./stop.sh worker && ./start.sh worker ``` +## Large payloads + +By default, payloads over 512KB will be offloaded to S3-compatible storage. If you don't provide the required env vars, runs with payloads larger than this will fail. + +For example, using Cloudflare R2: + +```bash +OBJECT_STORE_BASE_URL="https://..r2.cloudflarestorage.com" +OBJECT_STORE_ACCESS_KEY_ID="" +OBJECT_STORE_SECRET_ACCESS_KEY="" +``` + +Alternatively, you can increase the threshold: + +```bash +# size in bytes, example with 5MB threshold +TASK_PAYLOAD_OFFLOAD_THRESHOLD=5242880 +``` + ## Updating Once you have everything set up, you will periodically want to update your Docker images. You can easily do this by running the update script and restarting your services: From 1621b41d127684611da446c45960818e40c1f758 Mon Sep 17 00:00:00 2001 From: nicktrn <55853254+nicktrn@users.noreply.github.com> Date: Tue, 24 Sep 2024 13:03:09 +0100 Subject: [PATCH 4/8] additional features section and misc updates --- docs/open-source-self-hosting.mdx | 112 +++++++++++++++++------------- 1 file changed, 65 insertions(+), 47 deletions(-) diff --git a/docs/open-source-self-hosting.mdx b/docs/open-source-self-hosting.mdx index a06f436bfc..0f7b42402d 100644 --- a/docs/open-source-self-hosting.mdx +++ b/docs/open-source-self-hosting.mdx @@ -11,7 +11,7 @@ description: "You can self-host Trigger.dev on your own infrastructure." Self-hosting architecture -The self-hosting guide comes in two parts. The first part is a simple setup where you run everything on one server. In the second part, the webapp and worker components are split on two separate machines. +The self-hosting guide covers two alternative setups. The first options uses a simple setup where you run everything on one server. With the second option, the webapp and worker components are split on two separate machines. You're going to need at least one Debian (or derivative) machine with Docker and Docker Compose installed. We'll also use Ngrok to expose the webapp to the internet. @@ -55,7 +55,7 @@ Should the burden ever get too much, we'd be happy to see you on [Trigger.dev cl You will also need a way to expose the webapp to the internet. This can be done with a reverse proxy, or with a service like Ngrok. We will be using the latter in this guide. -## Part 1: Single server +## Option 1: Single server This is the simplest setup. You run everything on one server. It's a good option if you have spare capacity on an existing machine, and have no need to independently scale worker capacity. @@ -167,55 +167,104 @@ DEPLOY_REGISTRY_NAMESPACE= 3. Log in to Docker Hub both locally and your server. For the split setup, this will be the worker machine. You may want to create an [access token](https://hub.docker.com/settings/security) for this. ```bash -docker login -u +docker login -u docker.io ``` -4. Restart the services +4. Ensure the `docker-provider` container is logged in as well: + +```bash +docker exec -ti \ + trigger-docker-provider-1 \ + docker login -u docker.io +``` + +5. Restart the services ```bash ./stop.sh && ./start.sh ``` -5. You can now deploy v3 projects using the CLI with these flags: +6. You can now deploy v3 projects using the CLI with these flags: ``` npx trigger.dev@latest deploy --self-hosted --push ``` -## Part 2: Split services +## Option 2: Split services With this setup, the webapp will run on a different machine than the worker components. This allows independent scaling of your workload capacity. ### Webapp setup -All steps are the same as in Part 1, except for the following: +All steps are the same as for a single server, except for the following: -1. Run the start script with the `webapp` argument +1. **Startup.** Run the start script with the `webapp` argument ```bash ./start.sh webapp ``` -2. Tunnelling is now _required_. Please follow the tunnelling section from above. +2. **Tunnelling.** This is now _required_. Please follow the [tunnelling](/open-source-self-hosting#tunnelling) section. ### Worker setup -1. Copy your `.env` file from the webapp to the worker machine +1. **Environment variables.** Copy your `.env` file from the webapp to the worker machine: ```bash # an example using scp scp -3 root@:docker/.env root@:docker/.env ``` -2. Run the start script with the `worker` argument +2. **Startup.** Run the start script with the `worker` argument ```bash ./start.sh worker ``` -2. Tunnelling is _not_ required for the worker components. +3. **Tunnelling.** This is _not_ required for the worker components. + +4. **Registry setup.** Follow the [registry setup](/open-source-self-hosting#registry-setup) section but run the last command on the worker machine - note the container name is different: + +```bash +docker exec -ti \ + trigger-worker-docker-provider-1 \ + docker login -u docker.io +``` + +## Additional features + +### Large payloads + +By default, payloads over 512KB will be offloaded to S3-compatible storage. If you don't provide the required env vars, runs with payloads larger than this will fail. + +For example, using Cloudflare R2: + +```bash +OBJECT_STORE_BASE_URL="https://..r2.cloudflarestorage.com" +OBJECT_STORE_ACCESS_KEY_ID="" +OBJECT_STORE_SECRET_ACCESS_KEY="" +``` + +Alternatively, you can increase the threshold: -## Checkpoint support +```bash +# size in bytes, example with 5MB threshold +TASK_PAYLOAD_OFFLOAD_THRESHOLD=5242880 +``` + +### Version locking + +There are several reasons to lock the version of your Docker images: +- **Backwards compatibility.** We try our best to maintain compatibility with older CLI versions, but it's not always possible. If you don't want to update your CLI, you can lock your Docker images to that specific version. +- **Ensuring full feature support.** Sometimes, new CLI releases will also require new or updated platform features. Running unlocked images can make any issues difficult to debug. Using a specific tag can help here as well. + +By default, the images will point at the latest versioned release via the `v3` tag. You can override this by specifying a different tag in your `.env` file. For example: + +```bash +TRIGGER_IMAGE_TAG=v3.0.4 +``` + +### Checkpoint support This requires an _experimental Docker feature_. Successfully checkpointing a task today, does not @@ -225,14 +274,14 @@ scp -3 root@:docker/.env root@:docker/.env Checkpointing allows you to save the state of a running container to disk and restore it later. This can be useful for long-running tasks that need to be paused and resumed without losing state. Think fan-out and fan-in, or long waits in email campaigns. -The checkpoints will be pushed to the same registry as the deployed images. Please see the [Registry setup](#registry-setup) section for more information. +The checkpoints will be pushed to the same registry as the deployed images. Please see the [registry setup](#registry-setup) section for more information. -### Requirements +#### Requirements - Debian, **NOT** a derivative like Ubuntu - Additional storage space for the checkpointed containers -### Setup +#### Setup Underneath the hood this uses Checkpoint and Restore in Userspace, or [CRIU](https://github.com/checkpoint-restore/criu) in short. We'll have to do a few things to get this working: @@ -289,25 +338,6 @@ echo "FORCE_CHECKPOINT_SIMULATION=0" >> .env ./stop.sh worker && ./start.sh worker ``` -## Large payloads - -By default, payloads over 512KB will be offloaded to S3-compatible storage. If you don't provide the required env vars, runs with payloads larger than this will fail. - -For example, using Cloudflare R2: - -```bash -OBJECT_STORE_BASE_URL="https://..r2.cloudflarestorage.com" -OBJECT_STORE_ACCESS_KEY_ID="" -OBJECT_STORE_SECRET_ACCESS_KEY="" -``` - -Alternatively, you can increase the threshold: - -```bash -# size in bytes, example with 5MB threshold -TASK_PAYLOAD_OFFLOAD_THRESHOLD=5242880 -``` - ## Updating Once you have everything set up, you will periodically want to update your Docker images. You can easily do this by running the update script and restarting your services: @@ -354,18 +384,6 @@ git stash pop ./stop.sh && ./start.sh ``` -## Version locking - -There are several reasons to lock the version of your Docker images: -- **Backwards compatibility.** We try our best to maintain compatibility with older CLI versions, but it's not always possible. If you don't want to update your CLI, you can lock your Docker images to that specific version. -- **Ensuring full feature support.** Sometimes, new CLI releases will also require new or updated platform features. Running unlocked images can make any issues difficult to debug. Using a specific tag can help here as well. - -By default, the images will point at the latest versioned release via the `v3` tag. You can override this by specifying a different tag in your `.env` file. For example: - -```bash -TRIGGER_IMAGE_TAG=v3.0.4 -``` - ## CLI usage This section highlights some of the CLI commands and options that are useful when self-hosting. Please check the [CLI reference](/cli-introduction) for more in-depth documentation. From 8d20f06325b9e11ef8b4075d030dfd80dc34929b Mon Sep 17 00:00:00 2001 From: nicktrn <55853254+nicktrn@users.noreply.github.com> Date: Tue, 24 Sep 2024 13:03:28 +0100 Subject: [PATCH 5/8] troubleshooting section --- docs/open-source-self-hosting.mdx | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/docs/open-source-self-hosting.mdx b/docs/open-source-self-hosting.mdx index 0f7b42402d..4ea461820c 100644 --- a/docs/open-source-self-hosting.mdx +++ b/docs/open-source-self-hosting.mdx @@ -384,6 +384,30 @@ git stash pop ./stop.sh && ./start.sh ``` +## Troubleshooting + +- **Deployment fails at the push step.** The machine running `deploy` needs registry access: + +```bash +docker login -u +# this should now succeed +npx trigger.dev@latest deploy --self-hosted --push +``` + +- **Prod runs fail to start.** The `docker-provider` needs registry access: + +```bash +# single server? run this: +docker exec -ti \ + trigger-docker-provider-1 \ + docker login -u docker.io + +# split webapp and worker? run this on the worker: +docker exec -ti \ + trigger-worker-docker-provider-1 \ + docker login -u docker.io +``` + ## CLI usage This section highlights some of the CLI commands and options that are useful when self-hosting. Please check the [CLI reference](/cli-introduction) for more in-depth documentation. From 24ae17c86cfeeebdf4447dc7ffa939250d30f2e6 Mon Sep 17 00:00:00 2001 From: nicktrn <55853254+nicktrn@users.noreply.github.com> Date: Tue, 24 Sep 2024 14:16:23 +0100 Subject: [PATCH 6/8] add auth section --- docs/open-source-self-hosting.mdx | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/docs/open-source-self-hosting.mdx b/docs/open-source-self-hosting.mdx index 4ea461820c..30565c0441 100644 --- a/docs/open-source-self-hosting.mdx +++ b/docs/open-source-self-hosting.mdx @@ -264,6 +264,28 @@ By default, the images will point at the latest versioned release via the `v3` t TRIGGER_IMAGE_TAG=v3.0.4 ``` +### Auth options + +By default, magic link auth is the only login option. If the `RESEND_API_KEY` env var is not set, the magic links will be logged by the webapp container and not sent via email. + +All email addresses can sign up and log in this way. If you would like to restrict this, you can use the `WHITELISTED_EMAILS` env var. For example: + +```bash +# every email that does not match this regex will be rejected +WHITELISTED_EMAILS="authorized@yahoo\.com|authorized@gmail\.com" +``` + +It's currently impossible to restrict GitHub OAuth logins by account name or email like above, so this method is _not recommended_ for self-hosted instances. It's also very easy to lock yourself out of your own instance. + +Only enable GitHub auth if you understand the risks! We strongly advise you against this. + +Your GitHub OAuth app needs a callback URL `https:///auth/github/callback` and you will have to set the following env vars: + +```bash +AUTH_GITHUB_CLIENT_ID= +AUTH_GITHUB_CLIENT_SECRET= +``` + ### Checkpoint support From e5e71423ff05b3e0bd03b6098bb2717bffbe5a6b Mon Sep 17 00:00:00 2001 From: nicktrn <55853254+nicktrn@users.noreply.github.com> Date: Wed, 25 Sep 2024 11:08:07 +0100 Subject: [PATCH 7/8] skip default pr checks for docs changes --- .github/workflows/pr_checks.yml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/.github/workflows/pr_checks.yml b/.github/workflows/pr_checks.yml index 535ee9db6c..55bf20ad51 100644 --- a/.github/workflows/pr_checks.yml +++ b/.github/workflows/pr_checks.yml @@ -3,6 +3,8 @@ name: 🤖 PR Checks on: pull_request: types: [opened, synchronize, reopened] + paths-ignore: + - "docs/**" concurrency: group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }} From 092c3e6d3c132734e09855a465db5ec7cd963bc1 Mon Sep 17 00:00:00 2001 From: nicktrn <55853254+nicktrn@users.noreply.github.com> Date: Wed, 25 Sep 2024 11:32:42 +0100 Subject: [PATCH 8/8] minor changes --- docs/open-source-self-hosting.mdx | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/open-source-self-hosting.mdx b/docs/open-source-self-hosting.mdx index 30565c0441..ecb2b430a9 100644 --- a/docs/open-source-self-hosting.mdx +++ b/docs/open-source-self-hosting.mdx @@ -11,7 +11,7 @@ description: "You can self-host Trigger.dev on your own infrastructure." Self-hosting architecture -The self-hosting guide covers two alternative setups. The first options uses a simple setup where you run everything on one server. With the second option, the webapp and worker components are split on two separate machines. +The self-hosting guide covers two alternative setups. The first option uses a simple setup where you run everything on one server. With the second option, the webapp and worker components are split on two separate machines. You're going to need at least one Debian (or derivative) machine with Docker and Docker Compose installed. We'll also use Ngrok to expose the webapp to the internet. @@ -170,7 +170,7 @@ DEPLOY_REGISTRY_NAMESPACE= docker login -u docker.io ``` -4. Ensure the `docker-provider` container is logged in as well: +4. Required on some systems: Run the login command inside the `docker-provider` container so it can pull deployment images to run your tasks. ```bash docker exec -ti \