Deploying Dagster on AWS ECS with AWS Copilot

[ammerzon]

Hey Dagster community!

There is already a tutorial on how to deploy Dagster on AWS ECS: Deploying in ECS with the corresponding example project. Does anyone have experience deploying it this way using AWS Copilot? Respectively, do you happen to know if this is possible at all?

[ammerzon]

I have now deployed Dagster with AWS Copilot, so it is possible. If anyone needs help, please ping me.

[SSchneider22]

@ammerzon
I am a beginner in AWS Copilot, I am having trouble deploying Dagster to ECS using docker compose and I want to deploy Dagster to ECS with AWS Copilot.

I have two questions.

I would like to know the code in the Dockerfile.
What commands should I actually execute using AWS Copilot CLI?

[ammerzon]

To deploy the three services use the following commands:

# Init (only needed once)
copilot app init dagster
copilot svc init -n user-code
copilot svc init -n daemon
copilot svc init -n webserver
copilot env init -n prod
# Deploy
copilot env deploy --name prod
copilot deploy -e prod -n user-code
copilot deploy -e prod -n daemon
copilot deploy -e prod -n webserver

Regarding the Dockerfile I used a combination of the Poetry Dockerfile and the ECS deploy Dockerfile.

[SSchneider22]

Thank you for your answer!

I will try it with what you answered.

[Jhakir-sal]

Can you share your Dockerfile?

[ammerzon]

@Jhakir-sal Unfortunately, my Dockerfile is quite customized and includes several components that likely aren't relevant to your use case. However, the base I started from was the following file: https://github.com/dagster-io/dagster/blob/79ae041145d3e994e92e2c496073acab9956569a/examples/deploy_ecs/Dockerfile

[Suman5x]

Hello @ammerzon can you help me with the copilot tutorial?

[ammerzon]

Have you tried the template files I provided? 😁

[Suman5x]

Nope i have tried the docs and the docker compose push call is still running from an hour.

[ammerzon]

I haven't tried the Docker compose approach, so I can't help you with that, but you can use the provided ZIP if you want.

[RobertCastagna]

Hi @ammerzon, I'm using your templated structure and have (tried to write) written my own Dockerfile for it. I'm getting an error:

X execute Load Balanced Web Service init: check if docker engine is running: unmarshal docker info message: invalid character 'i' in literal false (expecting 'l')

after running :

.\copilot init --app app-name --name dagster --type "Load Balanced Web Service" --dockerfile "./src/Dockerfile" --port XX --deploy (ref: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/getting-started-aws-copilot-cli.html)

I'm running this command from the root of the project (inside directory that contains the folders src and dagster-copilot). Any idea if this is an issue with my Dockerfile or Docker application?

Docker version 25.0.3
aws-cli/2.0.21
Windows 10

edit: docker engine is running, tested with docker ps showing other actively running containers

[ammerzon]

This is a bug in the AWS Copilot CLI. Here you can find a workaround: aws/copilot-cli#5656 (comment)