Dynamicaly load dagster project modules and components

[milicevica23]

TLDR: The Dagster project loading should provide a mechanism to dynamically load Python modules where dagster definitions are defined and logically ignore components/modules

Let me first explain a general idea of what we do right now. Then, I will tackle some problems and, in the end, one possible solution

We are a platform team and want to make a template project for our company, which is then used in different teams across companies. We use https://cruft.github.io/cruft/ for templating https://cruft.github.io/cruft/, and the core of the project is dagster with other parts that are important for general infrastructure around the project
You can see more why we do this here: https://www.youtube.com/watch?v=GX508If6ol0&t=2625s
And also going more into what one project has here: https://www.youtube.com/watch?v=u5igKtEiKr8&t=24s

In the beginning, we were building a template project for one type of user group—data warehousing. Over time, data scientists joined, and the template project started to include a lot of stuff unrelated to the other user group.

Imagine one project having following and with the following structure

Over time, the user groups started to diverge in their needs, and the template project could no longer follow and fit into the project instance. And it started to be hard to update a project instance like the following picture

Data scientist started to delete dbt project because it was a overhead for the dependencies and compilation of the dbt project and loading of the dagster project started to take longer then needed
This is not ideal because maybe in the future they would need this peace of the code but right now they see it as an overhead and the question is how to solve this?

My thoughts:
Ideally there would be some way to define I want to load custom and bi module, and dagster under the hood would dynamically load those modules

You can imagine something similar as
pip install "fastapi[standard]"
where you dont install/start everything what fastapi has to offer but rather peaces of the code which you really need

In our case we use pixi package manager and use also multienvironment whe we can define a set of the dependencies needed for this feature https://pixi.sh/dev/workspace/multi_environment/

I could imagine having in the recursive loading functions a parameter which defines what submodules of the defs folder can be loaded in the code location

pixi enviroment definition file would look something like

[environments]
prod = {features = ["dbt"], solve-group = "prod"}

[feature.dbt.dependencies]
dbt-bigquery= "*"

[feature.bi.dependencies]
dagster-tableau = "*"

and definitions.py something like

import dagster as dg
import my_existing_project.defs

features = env.get_features() -> here comes just dbt because above it is defined just dbt 
defs = dg.components.load_defs(my_existing_project.defs, include_modules = features )

The idea is to dynamically load some modules which hold the definitions and therefore, make project usage simpler and more modular

[schrockn]

Is the primary objective here to be able to use a single template for all the different project variants? Or do you need more flexibility than that?

[milicevica23]

I am not sure if i understand what more flexibility would mean in this case?

My main goals were:

1. Providing one single template with all features and giving selection posibility
2. Reduce python environment size for ingest we use spark -> 1gb java and spark
3. Reduce speed for code location loading if feature not needed -> e.g dbt manifest parsing in local development or some not needed general imports which are in not used modules
   In debuging/development scenario you want to focus just on one module/component and load just that one

[schrockn]

I think there are three approaches here, some of which would require some modest framework changes.

Options:

1. Create separate code locations. I think this is the simplest option and the is goal is for built-in project scaffolding and the option for multi-project layout but it sounds like you don't want to do that.

2. Have commands that target subsets of defs. This is on our roadmap, but we would like to add support for dg to be able to target specific submodules for commands. dg list defs --target path/to/your/submodule and it would list the definitions that reside in defs/path/to/your/submodule.

3. Add framework support for placing arbitrary properties/tags in the context passed to build_defs and attach the pixi environment information to that. We have a component at every layer of the defs hierarchy and that would need to be customized to conditionally load certain submodules based on this contextual information.

4. Access the pixi-injected context (looks like it is available from the global environment variables) directly within build_defs

[milicevica23]

1. We dont want to do one time shot scaffolding but rather give something like parent project which provides standard components and you also always can bring your own
2. Yes this is something what i was thinking: that you can target or filter some submodules but it would be also very important that i can do this programatically in recursive defs load function in order that we can somehow parse the context and provide to the loading function
   3 & 4. This could be interesting but i think that dagster should not care about how and what is in the python environment but rather just provide a mechanism to subselect and not load some modules in the python interpreter. Dagster should stay package manager/environment indipendend