Would you consider integration with the component format of Kubeflow Pipelines? #21
Description
Kubeflow Pipelines is built on top of Argo, but brings additional features like Python-based SDK for component and pipeline authoring.
One important feature of the KFP SDK is the concept of reusable components.
The component author can create and share their component once and then many pipeline authors can load and use those components. The format for components is declarative and language-independent. They look a bit similar to Argo's templates, but are smaller, easier to write and have several features that make writing components easier. The component library also provides ways to create components from user-provided python functions (somewhat similar idea to closure) or Airflow operators. There is even a way to create sub-DAG components based on python functions describing the workflow.
KFP components are a popular feature and I've seen several hundreds of them in the public GitHub alone. Even big companies are creating their custom components based on the format.
Some examples that demonstrate the usage of the component library: Creating components from command-line programs Data passing in python components
Although currently the component library is part of the KFP SDK, it's essentially a standalone python module that does not have any dependency on the rest of KFP SDK. If you're interested, we can think of extracting it as a separate library that can be imported without any other code. The component library has well-defined extension points that allow integrating and bridging it with any other orchestration DSL library (KFP, Tekton, TFX, Airflow, Argo-DSL).
What do you think about this integration proposal?