Examples for k8s configuration

[JumpWill]

I have deployed dagster on my k8s ， successfully run a simple job and sheduler，
but do not know how to define a job pod on k8s for a dagter job ,for example use a yaml .
I just found the url https://github.com/dagster-io/dagster/tree/master/examples/deploy_k8s.
is there more document about how to define job pod for dagter job?

[garethbrickman]

Check out Customizing your Kubernetes Deployment

[JumpWill]

@garethbrickman
i am trying to use git-sync to update my repo code in time，
now i just know that clicking the reload button works well with git-sync image,
can i add some args for the commnad 'dagster code-server -h 0.0.0.0 -p 3030 --python-file xxx' to let the grpc server voluntarily reload when code changes

[JumpWill]

i just get info from https://discuss.dagster.io/t/18809556/how-can-i-reload-definitions-from-local-development-environm,
i can not get more info from dagster code-server -h

root@task-repo-59f4b4449f-tlngr:/repo# 
root@task-repo-59f4b4449f-tlngr:/repo# dagster code-server -h
Usage: dagster code-server [OPTIONS] COMMAND [ARGS]...

  Commands for working with Dagster code servers.

Options:
  -h, --help  Show this message and exit.

Commands:
  start  Start a code server that can serve metadata about a code location and launch runs.

[garethbrickman]

You can see the code for the code-server CLI command here: https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_cli/code_server.py

[kingsley-123]

I recently worked on comprehensive deployment documentation and encountered similar K8s configuration challenges when enhancing Dagster's concurrency and deployment guides.

For defining job pods in K8s with Dagster, here are the key configuration patterns that work:

1. Helm Chart Configuration (values.yaml):

dagster:
  instance:
    config:
      concurrency:
        runs:
          max_concurrent_runs: 10
          tag_concurrency_limits:
            - key: "kubernetes/namespace"
              value: "data-pipelines"
              limit: 5

dagsterUserDeployments:
  enabled: true
  deployments:
    - name: "your-job-deployment"
      image:
        repository: "your-registry/dagster-user-code"
        tag: "latest"
      dagsterApiGrpcArgs:
        - "--python-file"
        - "/path/to/your/definitions.py"
      port: 4000
      env:
        DAGSTER_POSTGRES_USER: ${POSTGRES_USER}
        DAGSTER_POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}

from dagster import define_asset_job, multiprocess_executor

configured_executor = multiprocess_executor.configured({
    "max_concurrent": 3,
    "tag_concurrency_limits": [
        {
            "key": "kubernetes/resource_type",
            "value": "memory_intensive", 
            "limit": 1
        }
    ]
})

k8s_job = define_asset_job(
    name="k8s_data_job",
    selection=["your_assets"],
    executor_def=configured_executor,
    tags={
        "dagster-k8s/config": {
            "container_config": {
                "resources": {
                    "requests": {"memory": "1Gi", "cpu": "500m"},
                    "limits": {"memory": "2Gi", "cpu": "1000m"}
                }
            }
        }
    }
)

The key is ensuring your dagster.yaml instance configuration, Helm chart settings, and job definitions all work together consistently.

You can see more complete examples and troubleshooting guidance in the enhanced deployment documentation I contributed: https://github.com/dagster-io/dagster/pull/30317.

What specific K8s configuration challenges are you facing? Are you having issues with pod resource allocation, environment variables, or something else? Happy to help with more targeted examples!