Remote storage for validation results and data docs in new version

[shabashaash]

As far as I understand, now (ver 1.*) it is not possible to save validation results and data docs in a remote location (for example, in S3 object storage)?
So, do I need to write and read everything manually (as indicated in the migration guide) using other providers (for example, S3Hook)?
Also do I understand correctly that when using GXValidateCheckpointOperator with FileDataContext, the default folder is /opt/airflow/dags? (the "configuration directory" mentioned in the documentation is a bit unclear to me)
And to load suits/checkpoints etc. I should place them is some predifined structure (like uncommited folder)?

[pankajkoti]

@joshua-stauffer could you please take a look at the above question and help answer? ☝🏽

[joshua-stauffer]

hi @shabashaash - that's correct, Great Expectations removed support for cloud store backends like S3 in v1, and so those backends are not available in Great Expectations Airflow Provider v1, either. If you're using a FileDataContext, you'll need to read/write your GX configuration from the cloud provider yourself.

Also do I understand correctly that when using GXValidateCheckpointOperator with FileDataContext, the default folder is /opt/airflow/dags?

When using GXValidateCheckpointOperator with FileDataContext, you are responsible for instantiating the FileDataContext, so there is no default folder. The GX config folder is controlled by the project_root_dir parameter to gx.get_context. There are some code examples for the configure_file_data_context function in the Great Expectations Airflow Provider migration guide:

import great_expectatations as gx
from great_expectations.data_context import FileDataContext

def configure_file_data_context() -> FileDataContext:
    # load your GX project from its remote source
    yield gx.get_context(
        mode="file",
        project_root_dir="./path_to_the_local_copy_of_your_existing_project"
    )
    # write your GX project back to its remote source

Within your gx configuration directory, ExpectationSuites and Checkpoints are persisted in gx/checkpoints and gx/expectations. In v1 GX, configuration can be done entirely through the Python API, so I would recommend that over manually editing the underlying JSON. There's more on the changes between GX 0.18 and 1.0 in the Great Expectations migration guide.

Let me know if you have more questions!