[Guide] Managing sensor timeouts

[dpeng817]

In this guide, we’ll discuss how to handle sensors that time out. Note: All suggestions apply to both Dagster Open Source (OSS) and Dagster Plus, unless specifically stated.

The default sensor timeout is 60 seconds, and sensors can hit that limit for a myriad of reasons. Most of the time, timeouts are the result of poorly optimized business logic or network issues, but occasionally, the business logic does require long evaluation cycles.

For these cases, we provide a mechanism to increase the default timeout of sensors, in both OSS and Plus. However, note that we consider changing the sensor timeout to be a last resort and the other options in this guide should be considered before attempting to change the timeout.

Finding the root of the timeout

The first step of addressing sensor timeouts is to understand why they are occurring. To do this, we recommend using py-spy, which allows you to profile your business logic and understand where the hot paths are. See this GitHub discussion for info about using py-spy.

Often, hot paths can manifest as calls to external APIs hanging or poorly optimized business logic. In the case of hanging external API calls, timing out is the appropriate failure mode. One might even consider making the sensor timeout shorter to improve resilience to these types of transient API errors.

Breaking up evaluations

If, after understanding the hot paths to your code, you don’t have external hanging API calls and there’s no clear place for optimization, the next step is to consider breaking up the expensive sensor evaluations into multiple steps. This can be done using sensor cursors. See our documentation for more info.

Using the right tool for the job (no pun intended)

The above advice is well and good if evaluations take slightly over 60 seconds, but it’s possible sensor evaluations need to take longer. Generally, if evaluations require more than 3 minutes to complete, a sensor may not be the correct tool for your needs.

Sensors are best used for frequently evaluating small amounts of data to kick off real work down the line. But if your sensors are running beyond the expected limit, then a scheduled job which sends run requests to engage in other work is more suited to your task.

Increasing the timeout

At this point in the guide, you should have identified that:

• Your sensor is timing out, but doesn’t need to run for over 3 minutes
• There are no obvious opportunities for code optimization
• Breaking up the sensor evaluation is not enough for you, or does not work for your use case

In this case, it’s time to consider increasing the sensor timeout.

Dagster Open Source (OSS)

In Dagster OSS, the timeout can be increased using the DAGSTER_SENSOR_GRPC_TIMEOUT_SECONDS environment variable. In OSS, the Dagster daemon sends requests directly to the user code servers. In both the daemon and code servers, the following can all be bottlenecks:

• CPU
• Memory
• Maximum request throughput

The number of workers available on the code server can be configured using the DAGSTER_GRPC_MAX_WORKERS environment variable. The number of workers available to the sensor daemon can be configured in the dagster.yaml sensor settings:

sensors:
  use_threads: true
  num_workers: 8

Dagster Plus

The sensor timeout can currently be changed as a deployment setting default_sensor_timeout. Follow the instructions here to access the settings for your deployment.

Set the default_sensor_timeout to the number of seconds that you wish your sensors to time out at. Note that this will affect all sensors in your deployment, including sensors added after this change.

Hybrid deployments

Hybrid deployments have a few additional options and considerations.

When you increase the timeout of your sensors, you are potentially placing more load on your user code deployment. In Hybrid deployments, it’s important to be aware of the health of your system and the mitigations available to you if increasing the timeout leads to any instability.

The above diagram describes how sensor requests flow through the system. They are generated in Dagster’s host cloud and then sent to the agent, which sends requests off to the code server running a particular sensor.

As sensor throughput increases, there’s the potential for bottlenecks anywhere in this flow. Each bottleneck is limited by 3 things:

1. The maximum concurrent requests that can be handled by the system component
2. The CPU limits of the component
3. The memory limits of the component

Bumping up against 1 or 2 usually manifests as increased lag times between sensor runs. Bumping up against 3 can lead to the underlying container being killed and restarted.

Monitoring the CPU and memory of the agent and code servers should be done using the underlying deployment service (k8s, ECS, etc), but is out of the scope of this guide. If you find that increasing the sensor timeout leads to issues, necessary action to increase your resource limits should be taken:

• For k8s, refer to this resource management guide.
• For ECS, we provide AgentMemory, AgentCpu, CodeServerMemory, and CodeServerCPU parameters on our Cloudformation agent template that allow you to easily configure your resource constraints.

Exposing agent and code server metrics in Dagster Plus Hybrid

Let’s turn our attention to handling the maximum request throughput of the agent/code server. With dagster >= 1.6.4, you can enable logging that exposes the current throughput of your agent and/or code server.

It is important to make sure that both your agent and your code server versions are running 1.6.4 or greater. The metrics collection feature is not backwards compatible with older versions of Dagster, and you could run into errors if both are not upgraded.

To check the version of an agent:

1. Click the Deployment tab in the top nav bar of the UI.
2. In the main window, click on the Agents tab.
3. Underneath the Labels header, you should see a number v1.x.x. This is the dagster version of your agent.

To check the version of a code server:

1. Click the Deployment tab in the top nav bar of the UI.
2. In the main window, click on the Code Locations tab.
3. Underneath the Actions header, there should be button with a dropdown, one for each code location. Click this dropdown icon, and select View dagster libraries.
4. The Version next to dagster is the version of your code server.

To check the version of a code server, click Deployment > Code locations tab:

After verifying your agent and code server versions meet the requirements, you can enable throughput logging:

• K8s: Add the following values to your Helm chart at the top level:

  agentMetrics:
    enabled: true
  codeServerMetrics:
    enabled: true

• ECS: Set the AgentMetricsEnabled and CodeServerMetricsEnabled settings to true.

• Docker: This functionality isn’t currently supported.

Once enabled, you should see metrics tracking the throughput of the agent in your agent logs. This includes not only the agent’s maximum throughput, but also each code server. In both cases, the request utilization metrics are provided in a JSON blob.

For the agent, the log looks like this:

dagster_cloud_agent - Current agent threadpool utilization: 0/20 threads

For the code server, the request utilization information must be parsed out of a larger JSON blob of information:

> user_code_launcher - Updated code server metrics for location sensor_test_87 in deployment prod: {'container_utilization': {'num_allocated_cores': 2, 'cpu_usage': 320.930756512, 'cpu_cfs_quota_us': -1.0, 'cpu_cfs_period_us': 100000.0, 'memory_usage': 150790144, 'memory_limit': 9223372036854771712, 'measurement_timestamp': 1708649283.031707, 'previous_cpu_usage': 320.823456119, 'previous_measurement_timestamp': 1708649221.99951}, 'request_utilization': {'max_concurrent_requests': 50, 'num_running_requests': 1, 'num_queued_requests': 0}, 'per_api_metrics': {'Ping': {'current_request_count': 1}}}

In the above code example, there are many fields that can be ignored. The important fields from the perspective of request throughput are:

• num_running_requests: tracks the number of currently running requests on the code server
• num_queued_requests: tracks number of enqueued requests on the code server (not currently running).
• max_concurrent_requests: the maximum number of requests which can run at any given time on the code server.

Increasing the agent’s maximum concurrent requests

It may turn out that you are reaching utilization for the number of requests your agent can serve. The default number of workers is dependent upon the number of cores available to your container. This number can be increased using the DAGSTER_CLOUD_AGENT_MAX_THREADPOOL_WORKERS environment variable. This should be set on the container running the agent itself.

Increasing the maximum concurrent requests for the code server

The default number of workers is dependent upon the number of cores available to your container. However, the number of concurrent requests can also be increased for code servers. This can be done with the DAGSTER_GRPC_MAX_WORKERS environment variable being set on startup in the container running the code server.

[dpeng817]

Answer in description

[hiimtu]

@dpeng817 Very interesting guide! I have some question on the definition of request to the agent and code server. Are these request to run jobs / sensors or does it also include other things?

For context, I'm looking to understand job queue and how to get visibility on the queue size in a hybrid deployment. Does the queuing / prioritization of jobs happen on the customer environment or dagster cloud?

Thanks in advance!

[dpeng817]

so the requests I'm referring to here are three things:

1. Requests to run a sensor
2. Requests to run a schedule
3. Requests for information about the code location, which power the UI.
   Requests to run a job are vended by dagster cloud, and then sent through the agent to a separate piece of machinery called a "run launcher", not the code server. but prioritization of jobs can be affected by a few different mechanisms in dagster. https://docs.dagster.io/guides/customizing-run-queue-priority#customizing-run-queue-priority could be a good starting point if you haven't seen yet.

[f-tremblay]

@dpeng817, thanks for the guide.

I can't find instructions on how to change deployment settings (i.e., default_sensor_timeout). We are on Dagster Cloud Serverless.
The link you provided seems to be unrelated.

The sensor timeout can currently be changed as a deployment setting default_sensor_timeout. Follow the instructions here to access the settings for your deployment.

[dpeng817]

Thanks for letting me know @f-tremblay. I just fixed the link.

[garethbrickman]

Note that if you're using an AutomationConditionSensorDefinition that is executing arbitrary Python code (so it has use_user_code_server=True enabled) then it will execute the Python code on your user code server instead of in the daemon.

This is the same place that your @sensor methods are evaluated. So the advice above and timeout settings etc. will apply.

[nicchristeller]

But if your sensors are running beyond the expected limit, then a scheduled job which sends run requests to engage in other work is more suited to your task

Is this not almost equivalent to a sensor?

If I understand correctly, the only place a "scheduled job" can send run requests is in its evaluation function, which is executed by the daemon service as is the sensor evaluation.

E.g. I have a sensor running every 10 minutes, and triggering a job if it wants. This suggestion is to instead have a schedule run every 10 minutes, and trigger a job if it wants. I don't really understand the functional difference - both do the evaluation outside of a run, and then trigger runs if they choose to.

[dpeng817]

Schedules evaluate within a grpc server request the same as a sensor. The runs kicked off from the schedule will be a "normal run" though. So the pattern to replicate schedule behavior would be to have a job with a very frequent schedule that just always kicks off an evaluation if none is in progress.

[nicchristeller]

Sorry to come back to this, but I'm having some real issues with the architecture here.

All I'm trying to do is to have some code that conditionally materializes an asset. This works with sensors (and could work with scheduled jobs as per this thread).

My problem is that I would really like for the code checking this condition to be isolated from the rest of the environment. Right now, I'm having issues with some sensors that used too much memory, and they bring down the whole code server - affecting every other asset.

As far as I can see, there's no way to conditionally materialize an asset from within a "run", and without a "run" there's no way to have this conditional code be isolated (in its own container, ideally). Please let me know if this isn't correct.

[dpeng817]

The bit that's not quite right here is that you can indeed kick off a materialization of an asset from within a run - a job is just running arbitrary user code; so you can, for example, use it to trigger dagster's own graphql API in order to kick off the materialization of the asset.

We provide the DagsterGraphQLClient for easy interaction with the graphql API via python: https://docs.dagster.io/guides/operate/graphql/graphql-client#overview you might need to create a job to wrap your asset to get this to work as you expect but should be entirely possible.

[nicchristeller]

The thing I was missing was the asset config output_required, so my multi-assets would materialise completely even when I only yielded a materialise result for a subset of them.

So the below is actually untrue (ignoring the fact that yes, I could always write some custom calls to the api) , but it took me a looong time to figure that out

there's no way to conditionally materialize an asset from within a "run"

[dpeng817]

apologies, I think I misunderstood exactly what you were asking. But yes, sounds like you have the right of it now. The documentation in this area is definitely a bit confusing.

[chrishiste]

@dpeng817 Hi, I think I found a bug for OSS.

I configured DAGSTER_SENSOR_GRPC_TIMEOUT_SECONDS through the Helm chart and it worked because I don't get the timeout error anymore but I still get another error.

It went from

dagster._core.errors.DagsterUserCodeUnreachableError: The sensor tick timed out due to taking longer than 60 seconds to execute the sensor function. One way to avoid this error is to break up the sensor work into chunks, using cursors to let subsequent sensor calls pick up where the previous call left off.

to

dagster._core.errors.DagsterUserCodeUnreachableError: Could not reach user code server. gRPC Error code: UNKNOWN

This happens right around the 60 seconds and every time. I tried restarting all my pods and refreshed the code location.

Any idea what could cause this? Thanks a lot

Full error:

dagster._core.errors.DagsterUserCodeUnreachableError: Unable to reach the user code server for sensor test_sensor. Sensor will resume execution once the server is available.
  File "/usr/local/lib/python3.10/site-packages/dagster/_daemon/sensor.py", line 270, in __exit__
    raise DagsterUserCodeUnreachableError(
The above exception was caused by the following exception:
dagster._core.errors.DagsterUserCodeUnreachableError: Could not reach user code server. gRPC Error code: UNKNOWN
  File "/usr/local/lib/python3.10/site-packages/dagster/_daemon/sensor.py", line 598, in _process_tick_generator
    yield from _evaluate_sensor(
  File "/usr/local/lib/python3.10/site-packages/dagster/_daemon/sensor.py", line 793, in _evaluate_sensor
    sensor_runtime_data = code_location.get_sensor_execution_data(
  File "/usr/local/lib/python3.10/site-packages/dagster/_core/remote_representation/code_location.py", line 970, in get_sensor_execution_data
    return sync_get_external_sensor_execution_data_grpc(
  File "/usr/local/lib/python3.10/site-packages/dagster/_api/snapshot_sensor.py", line 71, in sync_get_external_sensor_execution_data_grpc
    api_client.external_sensor_execution(
  File "/usr/local/lib/python3.10/site-packages/dagster/_grpc/client.py", line 567, in external_sensor_execution
    return self._query(
  File "/usr/local/lib/python3.10/site-packages/dagster/_grpc/client.py", line 205, in _query
    self._raise_grpc_exception(
  File "/usr/local/lib/python3.10/site-packages/dagster/_grpc/client.py", line 188, in _raise_grpc_exception
    raise DagsterUserCodeUnreachableError(
The above exception was caused by the following exception:
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNKNOWN
	details = "Exception calling application: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline Exceeded"
	debug_error_string = "UNKNOWN:Error received from peer  {created_time:"2025-04-11T19:42:13.328865541+00:00", grpc_status:4, grpc_message:"Deadline Exceeded"}"
>"
	debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"Exception calling application: <_InactiveRpcError of RPC that terminated with:\n\tstatus = StatusCode.DEADLINE_EXCEEDED\n\tdetails = \"Deadline Exceeded\"\n\tdebug_error_string = \"UNKNOWN:Error received from peer  {created_time:\"2025-04-11T19:42:13.328865541+00:00\", grpc_status:4, grpc_message:\"Deadline Exceeded\"}\"\n>", grpc_status:2, created_time:"2025-04-11T19:42:13.330025287+00:00"}"
>
  File "/usr/local/lib/python3.10/site-packages/dagster/_grpc/client.py", line 203, in _query
    return self._get_response(method, request=request_type(**kwargs), timeout=timeout)
  File "/usr/local/lib/python3.10/site-packages/dagster/_grpc/client.py", line 163, in _get_response
    return getattr(stub, method)(request, metadata=self._metadata, timeout=timeout)
  File "/usr/local/lib/python3.10/site-packages/grpc/_channel.py", line 1161, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.10/site-packages/grpc/_channel.py", line 1004, in _end_unary_response_blocking
    raise _InactiveRpcError(state)  # pytype: disable=not-instantiable

[ion-elgreco]

Same for me! I ended up just optimizing my code more, squeezing as much throughput

[dpeng817]

Hey folks sorry for the late reply.

I spent some time investigating this, and I think you're right that there's a bug here. The intended behavior is that you should just need to set DAGSTER_SENSOR_GRPC_TIMEOUT_SECONDS on the daemon, and that should just work. However, if you're using the codeServerArgs entrypoint to start your code servers, then it's getting lost in translation essentially when the request is actually made. A PR for a fix is here: #30697

Can you all confirm that you're using the codeServerArgs entrypoint? If not, there might be something else going on here.

In the short term, you can remediate by setting DAGSTER_SENSOR_GRPC_TIMEOUT_SECONDS on the daemon, and DAGSTER_GRPC_TIMEOUT_SECONDS on the code servers.

[rensoostenbachBL]

Hi @dpeng817

We use dagsterApiGrpcArgs:, for which I see the following documentation in the Helm chart:

The dagsterApiGrpcArgs key can also be replaced with codeServerArgs to use a new
experimental dagster code-server start command instead of dagster api grpc, which
can reload its definitions from within the Dagster UI without needing to restart the
user code deployment pod.

Is using dagsterApiGrpcArgs the same as using codeServerArgs entrypoint?

The PR looks like it could make it all cleaner, thanks!

We have 10+ code locations, and I was wondering if you can give insights into whether it is advisable to have a single timeout defined in the daemon, or if you think it should be defined per code location.

[dpeng817]

@rensoostenbachBL
Yea I think this is a scenario where our flexibility just isn't ideal yet. You need to set the timeout in the daemon, because without it being set, the request will time out client side. So unfortunately you're kind of locked in (at least right now) to setting the timeout somewhat globally across code locations. One feature I could imagine is setting the timeout on the sensor itself:

@sensor(timeout=300)
def my_sensor(...):
    ...

Or in the code location config:

# locations.yaml
locations:
    - name: my_location
       default:sensor_timeout: 60 

But yea that's just not possible as of yet. If you think it would be valuable, feel free to throw up a github issue.

[chrishiste]

I have codeServerArgs set on my side