My Dagster code is hanging or taking longer than I expect to execute. How can I troubleshoot?

[gibsondan]

• An op is taking longer than I would expect to run, how can I find the bottleneck?
• A dagster run is hanging and I want to know where it's stuck
• A code server has become unresponsive but doesn't seem to be memory or CPU-bound, why is it stuck?

[gibsondan]

We recommend that you use py-spy to profile your code.

For hanging code, py-spy dump can give you a dump of each thread, which usually makes it immediately clear where the hang is happening.

For slow code, py-spy record can produce file that gives you a flame graph of where the process is spending the most time. (We recommend py-spy record -f speedscope --idle to produce speedscope profiles, and to include idle CPU time in the results)

Note that py-spy usually requires elevated permissions in order to run.

For example, to run py-spy locally and understand why definitions are taking a long time to import:

sudo py-spy record -f speedscope --idle -- dagster definitions validate

--

A typical workflow for generating a py-spy dump for a hanging run in Kubernetes:

a) Set up your Dagster deployment so that each run pod is using a security context that can run py-spy https://github.com/benfred/py-spy#how-do-i-run-py-spy-in-kubernetes

For example, you can set this in your dagster_cloud.yaml file for your code location if you're running a Kubernetes agent to make both your code servers and run pods able to work with py-spy:

locations:
  - location_name: cloud-examples
    image: dagster/dagster-cloud-examples:latest
    code_source:
      package_name: dagster_cloud_examples
    container_context:
      k8s:
        server_k8s_config: # Raw kubernetes config for code servers launched by the agent
          container_config:
            securityContext:
              capabilities:
                add:
                  - SYS_PTRACE
        run_k8s_config: # Raw kubernetes config for runs launched by the agent
          container_config:
            securityContext:
              capabilities:
                add:
                  - SYS_PTRACE

Or if you're using the Dagster open-source helm chart, you can configure the run launcher to launch each run with

runLauncher:
  type: K8sRunLauncher
  config:
    k8sRunLauncher:
      runK8sConfig:
        containerConfig:
          securityContext:
            capabilities:
              add:
                - SYS_PTRACE

(Note that this gives the pod elevated permissions - check with your cluster admins to make sure this is an acceptable change to make temporarily)

See https://docs.dagster.io/dagster-cloud/deployment/agents/kubernetes/configuration-reference#per-location-configuration (Cloud) and https://docs.dagster.io/deployment/guides/kubernetes/customizing-your-deployment#per-job-or-per-op-kubernetes-configuration (OSS) for more information on how to apply these types of customizations to your Kubernetes pods.

b) Launch a run and wait until it hangs

c) Check the event logs for the run to find the run pod, then kubectl exec into the pod to run py-spy:

kubectl exec -it <pod name here> /bin/bash

d) install py-spy, then run it:

pip install py-spy
py-spy dump --pid 1

This should output a dump of what each thread in the process is doing.

[abi-mutinex]

@gibsondan if I want to see what's taking my code location ~600s to load, can I use this? Where would I add it?

[babaMar]

I get a CreateContainerConfigError when I add that securityContext in my Helm values.

This is what worked for me:

runLauncher:
  type: K8sRunLauncher
  config:
    k8sRunLauncher:
      securityContext:
        capabilities:
          add:
            - SYS_PTRACE

so not inside runK8sConfig.containerConfig.
Also, py-spy record -f speedscope --idle --pid 1 is crashing the pod.

[geoHeil]

How would this work for the docker based deployment?
I I need to roll out py-spy by default?
Or would I manually have to connect to the spun up container and install it on the fly?

Would it be possible to reuse the https://dagster.slack.com/archives/C02LJ7G0LAZ/p1686306297612339 special tag to launch the run in a persistently running container (where I can prepare the install of pyspy)?

[gibsondan]

py-spy has instructions for running in docker as well: https://github.com/benfred/py-spy#how-do-i-run-py-spy-in-docker

this would work similarly as the k8s instructions above but with something like:

locations:
  - location_name: cloud-examples
    image: dagster/dagster-cloud-examples:latest
    code_source:
      package_name: dagster_cloud_examples
    container_context:
      docker:
        container_kwargs:
          cap_add:
            - SYS_PTRACE

[yamhirotoCS]

Hi @gibsondan , I am trying to run py-spy in the user_code container to understand why a job container takes a lot of time (once ready on AWS) , to get started before actually running the job itself. the loading of the code location within the job container seems to be the issue. do you know which process should I track with py-spy?

• python /opt/dagster/dagster_home/.venv/bin/dagster code-server start -h 0.0.0.0 -p 4000 -m bi_dagster
• python -m dagster api grpc --lazy-load-user-code --socket /tmp/tmpbbs_u4y6 --heartbeat --heartbeat-timeout 30 --fixed-server-id 7dbfd3b4-11b1-45e6-a022-46faa7fec5c3 --log-level info --location-name bi_dagster --container-image dagster/user_code -m bi_dagster -d /opt/dagster/dagster_home
• python -c from multiprocessing.resource_tracker import main;

[gibsondan]

The code is actually being loaded in the second one (dagster api grpc)

[gibsondan]

That said, if you're launching a run using the default run launcher, that would be happening in some subprocess of that "dagster api grpc" process.

[yamhirotoCS]

Amazing, thanks for your quick response !
We are using dagster_aws.ecs as a run Launcher.
I will be checking :
python -m dagster api grpc --lazy-load-user-code --socket /tmp/tmpbbs_u4y6 --heartbeat --heartbeat-timeout 30 --fixed-server-id 7dbfd3b4-11b1-45e6-a022-46faa7fec5c3 --log-level info --location-name bi_dagster --container-image dagster/user_code -m bi_dagster -d /opt/dagster/dagster_home

as you suggested. thanks again !

[gibsondan]

If you're using the EcsRunLauncher, the code is being loaded in in the run inside an ECS task that is entirely separately from the code server. But what you describe there should still be useful to determine how long it takes the code to import, potentially.

[yamhirotoCS]

noted, I will check with the user_code first, as when we reduce the amount of assets to be loaded, not only the code location takes less time, but so does jobs executed locally. I will investigate locally and escalate to the ECS right after.
There seems to be noise with extra processes irrelevant to dagster, but I will dig start from there.

thanks !

[gibsondan]

Instructions for how to do this on a Hybrid Dagster+ ECS agent using ECS Exec: (requires the agent to be running at least version 1.9.0)

• Set user_code_launcher.config.enable_ecs_exec to true in the CloudFormation template for the agent: https://docs.dagster.io/dagster-plus/deployment/deployment-types/hybrid/amazon-ecs/configuration-reference#per-deployment-configuration
• Make sure that the task role being used for this code location has the needed IAM permissions to use ECS Exec:
• Use the ECS console to determine the cluster name and task ID of the code server task for this code location (it will be running in a service with the name of the code location in it)
• Install the AWS console and authenticate using a role with permissions to use ECS Exec
• Run a command like the following:

aws ecs execute-command --cluster <cluster name> --task <task ID> --container dagster --interactive  --command "/bin/bash"

You should then be shelled into the ECS task and be able to run the py-spy commands described above.