-
Notifications
You must be signed in to change notification settings - Fork 1.9k
Description
Bug summary
Hello, I am running Prefect self hosted on an EC2 instance in AWS. I have configured an AWS ECS work pool and am following this guide to create an AWS ECS based worker to pick up jobs.
The worker is correctly registered with my Prefect server and the work pool is active. The container image is prefecthq/prefect:3-latest
and the start up command is:
"command": [
"/bin/sh",
"-c",
"pip install prefect-aws && prefect worker start --pool aws-ecs-work-pool --type ecs"
],
When I execute a deployment, the job is successfully picked up by the worker, however it immediately fails when trying to submit the job to ECS for task execution. I get this error:
File "/usr/local/lib/python3.12/site-packages/botocore/session.py", line 424, in get_scoped_config
raise ProfileNotFound(profile=profile_name)
botocore.exceptions.ProfileNotFound: The config profile (prefect) could not be found
prefect.flow_runs.worker
Reported flow run '4ab3b0a2-5e29-4031-a621-f7159b46a880' as crashed: Flow run could not be submitted to infrastructure:
ProfileNotFound('The config profile (prefect) could not be found')
The ECS worker has an IAM role attached to it using the taskRoleArn and executionRoleArn configurations in the task definition
I have even tried hard coding the AWS_ACCESS_KEY_ID
and AWS_SECRET_ACCESS_KEY
environment variables of the task definition.
But no matter what I do, the execution always fails immediately with this error. It's like it's ignoring all of my AWS credentials config and always trying to find a profile with the name prefect
which doesn't exist and fails.
I also get the same error when running the worker on my local machine which is configured and authenticated to AWS. Running prefect worker start --pool "aws-ecs-work-pool"
and I get the same error.
Version info
Version: 3.4.10
API version: 0.8.4
Python version: 3.12.5
Git commit: fb979941
Built: Tue, Jul 22, 2025 07:23 PM
OS/Arch: darwin/arm64
Profile: xxx
Server type: server
Pydantic version: 2.11.7
Integrations:
prefect-aws: 0.5.13
Additional context
No response