Replace dagster_airbyte load_assets_from_airbyte_project #21230
adam-bloom
started this conversation in
Ideas
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
https://docs.dagster.io/integrations/airbyte#loading-airbyte-asset-definitions-from-yaml-config
This worked really well for a large number of airbyte connections. It is significantly more performant than
load_assets_from_airbyte_instance
and shortens dagster start times. (I had a use case where dagster startup just timed out on me when I tried to useload_assets_from_airbyte_instance
.)However, it relies on the
octavia
file structure. Airbyte has removedoctavia
completely, in favor of their new (if a little buggy) terraform provider. As such, we need a new way to define local airbyte connections. I was able to build a local implementation that wrapsbuild_airbyte_assets
easily enough, but I'm wondering if we should build an official way to extract information from terraform state files (I just duplicated). Additionally, we should probably update docs/deprecate the asset loader that depends onoctavia
Semi related to #19051
Beta Was this translation helpful? Give feedback.
All reactions