Skip to content

Add support for remote deployment setup in astro dev init #1928

@neel-astro

Description

@neel-astro

We want to add initial support for the client/server architecture split that the remote execution has as part of the Astro CLI's local project setup.
So, as part of the astro dev init command, we would want to introduce a --remote-execution-enabled boolean flag (default set to false).
When that flag is passed, the users would be prompted to enter the private registry hosting the client images for their Astro deployment. The private registry would be stored as part of the configuration in the local project. Note, we already have a notion of storing config for a local project, which looks like .astro/config.yaml in the filesystem, and the interaction is handled by the config package in the codebase. Storing it in config should allow users to modify the private registry endpoint down the line via the astro config set <config-name> <config-value> command.

$ astro dev init --remote-execution-enabled

Enter the remote Docker repository for the client image (leave blank if not known
but you will not be able to use the Astro CLI to deploy the image until configured)

Remote client image repository endpoint (e.g. quay.io/acme/my-deployment-image): quay.io/astronomer/ian-remote-eg

If you want to modify the remote repository down the line, use the `astro config set <config-name> <config-value>` command while being inside the local project

Note that we should expect the users to enter the image endpoint for the remote repository and not just the repository hostname, as not all registries support nested structure, and so it would not work if the CLI tries to add the image name on its own, which cannot be controlled by the user.

The dev init command with the additional flag should generate the following file structure for the local project. Note the additional client files

.astro/
  config.yaml
airflow_settings.yaml
dags/
Dockerfile
**Dockerfile.client**
include/
packages.txt
**packages-client.txt**
plugins/
README.md
requirements.txt
**requirements-client.txt**
tests/

The client Dockerfile should be initialized with the latest base runtime version of the client image or the runtime version passed in the input flag. We would have to think about a way to query available client images in the CLI. We could think about adding them to updates.astronomer.io

FROM images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-10-python....-base

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions