Version Control ECS Task Definition for `graphql-engine` in Moped #1707

frankhereford · 2025-10-27T20:08:40Z

Associated issues

This PR aims to close cityofaustin/atd-data-tech#24398.

Intent

This PR extends our graphql-engine CI such that it monitors the task definitions for our graphql-engine services in our Moped clusters, and installs and deploys them when they change.

Testing

As always, testing CI is a little different.

First, announce that you're testing to the devs. This is a one-person-testing-at-a-time type of PR.

When you are confident that it's just you working on this PR, then fire up your code editor, check out this branch and:

Edit staging.graphql-engine.ecs-td.json. We need to make a substantive change, not just white space. I suggest swithcing the "memory" key's value from 2048 to 4096. Remember to change it back when you're done please.
Commit this change into this feature branch and go ahead and push it up to GitHub.
Check out the actions in the GitHub web UI to see it work. Did it?
Confirm the changes in the AWS console in the ECS section. Looking good?

Hard-ish Mode

Did the workflow maintain the list of registered TDs to not be longer than the last 3?
Can you change the task definition in a non-substantive way (such as adding whitespace) and have the CI detect that no substantive changes have been made and not deploy the TD needlessly?
Did the CI properly serialize the tasks of deploying migrations & metadata independently of the TD deployment?
Did the CI show a meaningful diff showing just what had changed in the TD when it deployed it?
Does the CI know when it's not deployed a TD and know not to purge old TDs nor update the ECS service?
Does the CI detect a failed TD creation (such as if it's invalid or otherwise unsuitable) and not try to purge or restart anything?

⚠️ Before merger operations

There are two lines, about here & here, which need to be removed and reinstated respectively prior to merger.

Ship list

Code reviewed
Product manager approved
Product manager checked DB view dependencies
Product manager added to QA test script if applicable

…fied

…nition

…ification

…orkflow - Introduced functions for determining and validating ECS task definitions based on the branch name. - Enhanced the migration metadata deployment workflow to include steps for checking out code, configuring AWS credentials, and validating the ECS task definition. - Updated comments for clarity and organization in the helper script.

- Replaced the previous AWS CLI installation method with the official installation instructions for AWS CLI v2. - Commented out the old installation and configuration steps for clarity and future reference.

- Renamed the function for validating ECS task definitions to better reflect its purpose, changing it to register the task definition. - Updated the logic to register the ECS task definition using AWS CLI instead of validating it. - Adjusted comments for clarity regarding the registration process and its success or failure.

…ything

- Introduced a new function to update the ECS service with the latest task definition. - Added environment variables for the ECS cluster and service. - Enhanced logging to provide more context during the update process. - Updated the main task definition update process to include the service update step.

…on comparison results with emojis

netlify · 2025-10-27T20:08:45Z

✅ Deploy Preview for atd-moped-main ready!

Name	Link
🔨 Latest commit	`ebe3748`
🔍 Latest deploy log	https://app.netlify.com/projects/atd-moped-main/deploys/6903a4463f60850008f38f89
😎 Deploy Preview	https://deploy-preview-1707--atd-moped-main.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

frankhereford · 2025-10-27T20:25:32Z

.github/workflows/migrations-metadata-deployment.yml

+          echo "GR: ${GITHUB_REF}"
+          echo "PWD: $(pwd)"
+          source $(pwd)/.github/workflows/migrations-metadata-deployment-helper.sh
+          # run_migration_process


We want to avoid blasting main's database stack with the migrations and metadata in this branch as we dev and test, so this line is commented out. It needs to be commented back in just prior to merger (or as a direct push on main should we forget.)

frankhereford · 2025-10-27T20:27:42Z

.github/workflows/migrations-metadata-deployment.yml

+        run: |
+          curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
+          unzip awscliv2.zip
+          sudo ./aws/install --update


I was surprised to learn that we use the AWS cli version 1 in our old, existing CI here, but in retrospect, the timing is not too surprising. V2 came out in 2020. V1 is still supported, but V2 is the future.

Anyway, that's why I --update here.

frankhereford · 2025-10-27T20:37:32Z

moped-database/ecs_task_definitions/production.graphql-engine.ecs-td.json

H/T to @mddilley who spent some time with me looking as these task definitions until we were both satisfied that we were comfortable with them being released into the public. Thank you Mike 🙏

frankhereford · 2025-10-27T20:41:14Z

.github/workflows/migrations-metadata-deployment-helper.sh

+
+  # Extract just the taskDefinition object and remove AWS-managed fields
+  # These fields are added by AWS and shouldn't be compared
+  jq --sort-keys '.taskDefinition | del(.taskDefinitionArn, .revision, .status, .requiresAttributes, .compatibilities, .registeredAt, .registeredBy, .deregisteredAt, .tags)' \


Sort keys is important here to make the ordering of KV pairs in the JSON deterministic, because we actually compare these json blobs with diff as if they were code, and not by doing a == inside JS or python or what-have-you.

Anyway - with jq here, we strip out things that are intrinsic (such as those items supplied by AWS) and we only look at what we're actually defining.

An upside of doing it this way is that it becomes trivial for us to include the output of that diff in the logs, which is a super easy way for a dev to figure out what change caused the new task definition to be deployed into the service.

It's really cool to see how you solved this. How are you feeling about how future proof this is, and also if concerns about false positives due to minor formatting differences?

@frankhereford I might be way overthinking, but I'm curious if we could consider deploying the task definition if the version-controlled file changes at all instead of comparing to what is currently deployed. Since we will be reviewing any code changes in a PR, we could give ourselves responsibility for making sure that we understand the changes we are pushing to be deployed in the service (like we do right now during the launch parties).

I'm also thinking about needing to maintain this list of fields that are added by AWS in the future. I really could be worrying to much but wanted to get some discussion going on this in case I am overlooking something.

frankhereford · 2025-10-27T20:42:36Z

.github/workflows/atd_moped_database.yml

We're doing some renaming of some files here to give them names that better represent what they do, not what they act upon.

frankhereford · 2025-10-27T20:47:26Z

.github/workflows/migrations-metadata-deployment.yml

+    branches:
+      - main
+      - production
+      - frank/version-control-ecs-task-definitions


This line needs to be removed before merger, and it is what allows us to test this PR.

Copilot

Pull Request Overview

This PR implements version control for ECS task definitions for the graphql-engine service in Moped. The changes enable automatic monitoring and deployment of task definition updates through CI/CD, replacing manual task definition management with a declarative approach.

Key changes:

Adds new ECS task definition JSON files for staging and production environments
Introduces automated task definition deployment workflow that detects changes, registers updates, and manages service deployments
Consolidates and enhances the existing migration workflow to include task definition management

Reviewed Changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
moped-database/ecs_task_definitions/staging.graphql-engine.ecs-td.json	New task definition for staging graphql-engine service with 2GB memory allocation
moped-database/ecs_task_definitions/production.graphql-engine.ecs-td.json	New task definition for production graphql-engine service with 8GB memory allocation
.github/workflows/migrations-metadata-deployment.yml	Enhanced workflow that now handles both migrations/metadata and ECS task definition deployments
.github/workflows/migrations-metadata-deployment-helper.sh	New comprehensive helper script containing functions for migrations, task definition registration, cleanup, and service updates
.github/workflows/aws-moped-migrations-helper.sh	Removed legacy helper script, functionality moved to migrations-metadata-deployment-helper.sh
.github/workflows/atd_moped_database.yml	Removed legacy workflow, replaced by migrations-metadata-deployment.yml

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

.github/workflows/migrations-metadata-deployment.yml

.github/workflows/migrations-metadata-deployment-helper.sh

mddilley

I am releasing my comments before I test the workflow after John. 🥽 🧪 I mentioned you in one question that I'm still thinking through.

mddilley · 2025-10-30T17:13:37Z

moped-database/ecs_task_definitions/staging.graphql-engine.ecs-td.json

+            },
+            "memory": 2048,
+            "memoryReservation": 2048,
+            "mountPoints": [ ],


we don't have a team-wide formatter for json that we're set on so I want to get feedback from others. I noticed that when I ran "Prettify JSON" from VSCode it remove the spaces between these brackets. I don't think it is blocking the merge of this since we will always review a code diff when updating this file anyways.

Mostly wondering if there is a good extension that I'm missing out on (or yet another one that doesn't work in my editor like with markdown 🫠 😅)

mddilley · 2025-10-30T17:18:15Z

moped-database/ecs_task_definitions/staging.graphql-engine.ecs-td.json

I did a side-by-side of the JSON in staging in the AWS console and noticed some small differences with compatibilities and requiresAttributes not present in this file but then I looked at the docs and see that we should be validating for Fargate only and that Fargate doesn't support the requiresAttributes parameter anyways. Makes sense to remove since we've always used Fargate and TIL! 🙏

mddilley · 2025-10-30T17:32:45Z

moped-database/ecs_task_definitions/production.graphql-engine.ecs-td.json

+                {
+                    "name": "MOPED_API_ACTIONS_URL",
+                    "valueFrom": "arn:aws:ssm:us-east-1:295525487728:parameter/MOPED_PRODUCTION_FARGATE_HASURA_ACTIONS_URL"
+                },
+                {
+                    "name": "MOPED_API_APIKEY",
+                    "valueFrom": "arn:aws:ssm:us-east-1:295525487728:parameter/MOPED_PRODUCTION_FARGATE_HASURA_API_KEY"
+                }


I can make a spin-off issue snackoo to cover this but this reminds me that we have some cleanup to do in these secrets. I believe that these were used by our previous SQS-powered activity log events process.

mddilley · 2025-10-30T17:36:46Z

.github/workflows/migrations-metadata-deployment-helper.sh

+################################################################################
+
+#
+# Download the Hasura settings from the AWS Secrets Manager


thank you for cleaning up this last confusing reference to Zappa 😵‍💫🙏 and improving the comments ✨

mddilley · 2025-10-30T17:46:29Z

.github/workflows/migrations-metadata-deployment-helper.sh

+
+  # Extract just the taskDefinition object and remove AWS-managed fields
+  # These fields are added by AWS and shouldn't be compared
+  jq --sort-keys '.taskDefinition | del(.taskDefinitionArn, .revision, .status, .requiresAttributes, .compatibilities, .registeredAt, .registeredBy, .deregisteredAt, .tags)' \


@frankhereford I might be way overthinking, but I'm curious if we could consider deploying the task definition if the version-controlled file changes at all instead of comparing to what is currently deployed. Since we will be reviewing any code changes in a PR, we could give ourselves responsibility for making sure that we understand the changes we are pushing to be deployed in the service (like we do right now during the launch parties).

I'm also thinking about needing to maintain this list of fields that are added by AWS in the future. I really could be worrying to much but wanted to get some discussion going on this in case I am overlooking something.

johnclary

This so cool, Frank. My tests work exactly as expected. The diff logging is super fancy. It is great getting to learn some dev ops magic through your PRs 🙌

Glad to see your big bold reminder to re-enable the migrations job and remove this branch from the paths trigger 😉

🚢 🚢

johnclary · 2025-10-29T18:55:38Z

.github/workflows/migrations-metadata-deployment-helper.sh

+#
+# Deregisters old task definitions, keeping only the last 3 active ones
+#
+function cleanup_old_task_definitions() {


this is a really nice touch, and i'm mostly taking it for granted that this works as promised 👍

i see now that it does work! (with my latest revision the most recent of the three) ✨

@frankhereford one last thought on this. I wonder if we might want to increase to preserving maybe 10 of the most recent revisions. Reason being that although the latest task definition may install successfully, it is still possible that the definition has a faulty configuration that causes the container deployment to fail. If something goes sideways, it might be a relief to have a working task def available in AWS so we can recover quickly while we triage the issue. This would be excessive caution, of course, because we do have task definitions version controlled in Github as well.

johnclary · 2025-10-29T19:00:50Z

.github/workflows/migrations-metadata-deployment-helper.sh

+
+  # Extract just the taskDefinition object and remove AWS-managed fields
+  # These fields are added by AWS and shouldn't be compared
+  jq --sort-keys '.taskDefinition | del(.taskDefinitionArn, .revision, .status, .requiresAttributes, .compatibilities, .registeredAt, .registeredBy, .deregisteredAt, .tags)' \


It's really cool to see how you solved this. How are you feeling about how future proof this is, and also if concerns about false positives due to minor formatting differences?

frankhereford added 30 commits October 21, 2025 11:12

Add json TD specification for production and staging, from AWS unmodi…

ffab34a

…fied

Rename workflow and helper script, and enhance task names in job defi…

32173cd

…nition

Add placeholder job to migration metadata workflow for completion not…

21896fb

…ification

rename this file to match

daee945

remove header; there is a name directly below it with the same content

6a120da

disarm the migration job for development

8bc92ad

Update path for triggering files

3489c5f

Start to address boilerplate code

e6fd5e5

allow for this branch to trigger actions for dev

e3efa70

typo in file name

19b9a5b

Add AWS cli tooling

5b879d6

Use official installation instructions from AWS for cli

ecdaca6

V2 out of the ubuntu repo

f026504

Update AWS CLI installation method in deployment workflow

43d88f7

- Replaced the previous AWS CLI installation method with the official installation instructions for AWS CLI v2. - Commented out the old installation and configuration steps for clarity and future reference.

Add ECS task family environment variable to deployment helper script

bb46adf

Checking if file changed in this push

23c1cea

Sync up to latest graphql-engine

337196f

try a more robust approach to change detection

8b54352

Update workflow, but not TD, to test normalization technique

1828ec6

OK, this may be working -- bump the ram for example

ce0a54b

Add deregistration support

78b49b1

we need to make sure we're running the latest before we deregister an…

8fc4d06

…ything

Restore previous 2G memory allocation

91ce1f6

Allow deregistration of old TDs now

6710547

Enhance logging in deployment helper script to indicate task definiti…

1bc6344

…on comparison results with emojis

Reduce CPU allocation in ECS task definition from 512 to 256

fc87608

Got a free test of invalid TD's on that one, worked good!

c4ea1ab

frankhereford commented Oct 27, 2025

View reviewed changes

TD EOF Newlines

d21cc06

frankhereford self-assigned this Oct 27, 2025

frankhereford commented Oct 27, 2025

View reviewed changes

Get the version right now that dev is done

1f172a6

frankhereford requested a review from Copilot October 27, 2025 21:16

Copilot AI reviewed Oct 27, 2025

View reviewed changes

frankhereford requested review from chiaberry, johnclary, mateoclarke, mddilley, roseeichelmann and tillyw October 27, 2025 21:20

johnclary added 5 commits October 30, 2025 13:31

john testing: bump memory and memoryReservation to 4096

ce31eea

john testing: bump task memory to 4096

dea7f4d

john testing: revert everything

8439c60

john testing: add env var

93dc1d7

john testing: remove env var

ebe3748

mddilley reviewed Oct 30, 2025

View reviewed changes

johnclary approved these changes Oct 30, 2025

View reviewed changes

Version Control ECS Task Definition for graphql-engine in Moped #1707

Are you sure you want to change the base?

Version Control ECS Task Definition for graphql-engine in Moped #1707

Uh oh!

Conversation

frankhereford commented Oct 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Associated issues

Intent

Testing

Hard-ish Mode

⚠️ Before merger operations

Ship list

Uh oh!

netlify bot commented Oct 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for atd-moped-main ready!

Uh oh!

frankhereford Oct 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

frankhereford Oct 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

frankhereford Oct 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mddilley Oct 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mddilley left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mddilley Oct 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mddilley Oct 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

johnclary left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

johnclary Oct 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Version Control ECS Task Definition for `graphql-engine` in Moped #1707

Version Control ECS Task Definition for `graphql-engine` in Moped #1707

frankhereford commented Oct 27, 2025 •

edited

Loading

netlify bot commented Oct 27, 2025 •

edited

Loading

frankhereford Oct 27, 2025 •

edited

Loading

frankhereford Oct 27, 2025 •

edited

Loading

frankhereford Oct 27, 2025 •

edited

Loading

mddilley Oct 30, 2025 •

edited

Loading

mddilley left a comment •

edited

Loading

mddilley Oct 30, 2025 •

edited

Loading

mddilley Oct 30, 2025 •

edited

Loading

johnclary Oct 30, 2025 •

edited

Loading