chore(test): Support e2e cluster setup with Podman

[astefanutti]

What this PR does / why we need it:

Support setting up e2e cluster with Podman in addition to Docker.

Checklist:

• Docs included if any changes are user facing

[coveralls]

Pull Request Test Coverage Report for Build 18186567965

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

• For more information on this, see Tracking coverage changes with pull request builds.
• To avoid this issue with future PRs, see these Recommended CI Configurations.
• For a quick fix, rebase this PR at GitHub. Your next report should be accurate.

Details

• 0 of 0 changed or added relevant lines in 0 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage remained the same at 55.174%

---

Totals
Change from base Build 18045194431:	0.0%
Covered Lines:	1093
Relevant Lines:	1981

---

💛 - Coveralls

[jaiakash]

Hi @andreyvelich Can you please add the ok-to-test-gpu-runner label to this PR to verify if podman doesn't break the GPU E2E test.
For GPU E2E test, We are setting up the runtime to docker here, wanted to confirm that part.

trainer/docs/proposals/2432-gpu-testing-on-llm-blueprints/OCI VM/bootstrap.sh

Line 71 in b918411

sudo nvidia-ctk runtime configure --runtime=docker

[andreyvelich]

@astefanutti Did you add support for Podman to allow users run this E2Es locally, right?
IIUC, we still use Docker for our CI tests.

[astefanutti]

@astefanutti Did you add support for Podman to allow users run this E2Es locally, right?
IIUC, we still use Docker for our CI tests.

@andreyvelich you're right, it's meant to run E2Es locally. That being said @jaiakash rightly suggested to run the GPU E2E to make sure there is no regression.

[astefanutti]

@jaiakash should nvkind be only used to create the KinD cluster and then kind be used, or can it be used to "alias" kind entirely?

[jaiakash]

@jaiakash should nvkind be only used to create the KinD cluster and then kind be used, or can it be used to "alias" kind entirely?

Yes for creating the cluster, nvkind have to be used. Default kind binary wont work. Ref: https://github.com/NVIDIA/nvkind Once the cluster is created, we can use the regular kind commands as usual, since it behaves like a standard KinD cluster after provisioning.

One potential issue I foresee is that we're currently configuring the NVIDIA container runtime for Docker:

sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

If we switch the container runtime to Podman, this setup might not work as expected. I was looking into this guide: Podman GPU Support. I will check it tmr morning btw.

[astefanutti]

Yes for creating the cluster, nvkind have to be used. Default kind binary wont work. Ref: https://github.com/NVIDIA/nvkind Once the cluster is created, we can use the regular kind commands as usual, since it behaves like a standard KinD cluster after provisioning.

@jaiakash thanks, I've updated it accordingly.

One potential issue I foresee is that we're currently configuring the NVIDIA container runtime for Docker:

sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

If we switch the container runtime to Podman, this setup might not work as expected. I was looking into this guide: Podman GPU Support. I will check it tmr morning btw.

It would be nice if that would work but we don't have to switch the GH Actions E2E workflows to use Podman.
It's really meant for people / contributors that use Podman to run the tests locally.

[jaiakash]

Thanks for the edit. @astefanutti

[jaiakash]

My bad it got assigned to me, I wanted to just LGTM the PR not assign me.

/assign @astefanutti

[astefanutti]

@jaiakash thanks.

/assign @kubeflow/kubeflow-trainer-team

[tenzen-y]

Thank you 👍
/lgtm
/approve

[google-oss-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: tenzen-y

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [tenzen-y]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment