Kedro-Vizro integration

[antonymilne]

@antonymilne wrote:

I'd love to hear any more feedback/ideas you have for the kedro integration. At the moment it's pretty lightweight but I'd like to try and make it deeper.

@gtauzin wrote:

Actually, I've been thinking to open an issue on the kedro issue tracker to discuss this. Here are a few ideas:

1. How should I think about integrating my vizro dashboards in my kedro projects?

Somehow, if you use both together, your dahsboards are fed by kedro pipelines and take kedro dataset in. They are part of the same data science project, so it makes sense to me to deal with them together within a monorepos.

For now, I put them in a dashboards subfolder at the same level as pipelines

.
├── conf
├── data
├── docs
├── notebooks
├── pyproject.toml
├── README.md
├── src
│   └── my_kedro_project
│       ├── dashboards
│       │   ├── __init__.py
│       │   └── iris
│       ├── hooks.py
│       ├── __init__.py
│       ├── pipeline_registry.py
│       ├── pipelines
│       │   ├── data_processing
│       │   ├── data_science
│       │   ├── __init__.py
│       └── settings.py
└── tests

I organize dependencies using the optional-dependencies section of my pyproject.toml, e.g. I'll have one set of dependencies for "dashboards.iris". Note this relates to kedro-org/kedro#4147

2. If the vizro dashboards are "nodes" in your pipeline, then I'd love to be able to see them within my kedro viz visualization.

3. I would love to be able to use CLI to create new vizro apps within my kedro project the same way I create new pipelines: kedro vizro create my_dashboard

4. kedro has kedro-docker which helps containerize your pipelines, I'd love to have it work for vizro dashboards as well.

If there's more discussions on that topic, I'd love to follow them! Do you have any concrete plan to make the kedro/vizro integration? It seems to me a kedro plugin for vizro would be really nice.

[antonymilne]

Other comments from @gtauzin in #1001:

I guess I like to think about a vizro dashboard as something equivalent to a Kedro node. It takes a list of inputs and returns a list of outputs (e.g. in the case of a labeling dashboard). In practice, I would even like to visualize the inputs/outputs of my dashboards in kedro viz.

The CI is failing because I added a polars dataset to the catalog and polars is not included in the dependencies.

This is an interesting edge case because in the case where I have a large number of datasets in my catalog and I want to load a few of them (all pandas-based) for a vizro dashboards, I should not be forced to include all dependencies related to all datasets in my catalog. Does it make sense? I feel that datasets_from_catalog should take a list of dataset names as a parameter so we can control which ones are instantiated.