Kedro pipeline + dask dataframe integration

[sighingnow]

• deployment

  • align kedro with existing workflow yaml
  • for each workflow,
    • pre-launch vineyard + dask
  • lifecycle
  • adding launch vineyard to argo? as a prepare step.

• integration

  • VineyarDataset integrate with dask.dataframe (put/get)
  • spill to minio

• APIs

  • reuse kedro framework
  • make function take dask.dataframe/series....
  • catalog: VineyardDataset

[sighingnow]

/cc @wuyueandrew