Kubeflow as a Provider for multiple Llama Stack Components

[franciscojavierarceo]

Hi, all!

I, along with @andreyvelich, @terrytangyuan, @johnugeorge, and @juliusvonkohout from the Kubeflow Steering Committee wanted to reach out about a collaboration between Kubeflow and Llama Stack.

Kubeflow is an ecosystem of Kubernetes based components for each stage in the AI/ML Lifecycle with support for best-in-class open source tools and frameworks.

The Kubeflow community is interested in implementing our relevant components to Llama Stack as providers (e.g., #1319).

At a high level, we would like to incorporate the Kubeflow Trainer, KServe, Model Registry, Katib, Kubeflow Pipelines, and Feast.

We wanted to open a discussion to get feedback from the Llama Stack community and maintainers about their appetite for this. Kubeflow has a long history of serving production AI/ML use cases on Kubernetes and we think this would be a great opportunity to build on top of the tooling the Kubeflow community has built. For the Kubeflow community, we are excited to highlight the power of Kubeflow for scaling GenAI Applications using open source.

[booxter]

FYI prototype of KF Pipelines workflow-agnostic* job scheduler (both local and remote modes - so you can use the same provider code to run inline and remote - with a tiny adjustment) can be found here: https://github.com/opendatahub-io/llama-stack-provider-kfp-trainer It's using external providers infra. At the moment I don't have immediate plans or capacity to get this into mainline, but we may have this piece of code as a starter in case we need KFP in the future.

• e.g. could be applied to training, eval, sdg etc. as needed.

[franciscojavierarceo]

Awesome stuff @booxter!

@cdoern could you share some of the challenges you experienced and how this was helpful for you with post training?

FYI @raghotham @andreyvelich @szaher @anishasthana @HumairAK

[cdoern]

Sure, thanks for the tag. When trying to run distributed post-training without a sort of KF job scheduler that could run async and in the background, LLS times out after a short period of time:

 python3.11 train.py
Traceback (most recent call last):
  File "/Users/charliedoern/projects/Documents/llama-stack-provider-kft/venv/lib/python3.11/site-packages/httpx/_transports/default.py", line 101, in map_httpcore_exceptions
    yield
  File "/Users/charliedoern/projects/Documents/llama-stack-provider-kft/venv/lib/python3.11/site-packages/httpx/_transports/default.py", line 250, in handle_request
    resp = self._pool.handle_request(req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/charliedoern/projects/Documents/llama-stack-provider-kft/venv/lib/python3.11/site-packages/httpcore/_sync/connection_pool.py", line 256, in handle_request
    raise exc from None
  File "/Users/charliedoern/projects/Documents/llama-stack-provider-kft/venv/lib/python3.11/site-packages/httpcore/_sync/connection_pool.py", line 236, in handle_request
    response = connection.handle_request(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/charliedoern/projects/Documents/llama-stack-provider-kft/venv/lib/python3.11/site-packages/httpcore/_sync/connection.py", line 103, in handle_request
    return self._connection.handle_request(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/charliedoern/projects/Documents/llama-stack-provider-kft/venv/lib/python3.11/site-packages/httpcore/_sync/http11.py", line 136, in handle_request
    raise exc
  File "/Users/charliedoern/projects/Documents/llama-stack-provider-kft/venv/lib/python3.11/site-packages/httpcore/_sync/http11.py", line 106, in handle_request
    ) = self._receive_response_headers(**kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/charliedoern/projects/Documents/llama-stack-provider-kft/venv/lib/python3.11/site-packages/httpcore/_sync/http11.py", line 177, in _receive_response_headers
    event = self._receive_event(timeout=timeout)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/charliedoern/projects/Documents/llama-stack-provider-kft/venv/lib/python3.11/site-packages/httpcore/_sync/http11.py", line 217, in _receive_event
    data = self._network_stream.read(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/charliedoern/projects/Documents/llama-stack-provider-kft/venv/lib/python3.11/site-packages/httpcore/_backends/sync.py", line 126, in read
    with map_exceptions(exc_map):
  File "/opt/homebrew/Cellar/python@3.11/3.11.10/Frameworks/Python.framework/Versions/3.11/lib/python3.11/contextlib.py", line 158, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/Users/charliedoern/projects/Documents/llama-stack-provider-kft/venv/lib/python3.11/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc) from exc
httpcore.ReadTimeout: timed out

This looks like a basic http timeout from the kubeflow trainer specifically in this case and there is no LLS native way to gracefully handle this without using a job scheduler that is asynchronous + background friendly.

Tangentially related, I generally +1 this initiative and have a kubeflow trainer provider here: https://github.com/opendatahub-io/llama-stack-provider-kft which I'd love to generalize and put upstream as a training provider!