RESOURCE_EXHAUSTED or 413 Client Error with Having Multiple dbt Projects in a Single Code Location

[ToddArmstrong-IPG]

Hello -

We have several dbt projects to support various clients that we'd like to orchestrate from a single Dagster+ Code Location. Everything works fine up until we get to about 25 dbt projects. At this point we start getting errors like this...

dagster._core.errors.DagsterUserCodeUnreachableError: Could not reach user code server. gRPC Error code: RESOURCE_EXHAUSTED

grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.RESOURCE_EXHAUSTED
details = "Sent message larger than max (194508291 vs. 50000000)"
debug_error_string = "UNKNOWN:Error received from peer {grpc_message:"Sent message larger than max (194508291 vs. 50000000)", grpc_status:8, created_time:"2024-01-02T11:01:36.796016+08:00"}"
>

For example.

I have tried increasing DAGSTER_GRPC_MAX_RX_BYTES and DAGSTER_GRPC_MAX_SEND_BYTES, but this results in a time out when loading the Code Location.

We are also not generating the manifest at run time, they are pre-built and in our repo.

Does anyone have any recommendations for how to load many dbt projects in Dagster at once?

Thank you!