Synchronize trainer manifests 2.0 #3181

kunal-511 · 2025-07-01T18:12:45Z

Pull Request Template for Kubeflow Manifests

✏️ Summary of Changes

Describe the changes you have made, including any refactoring or feature additions.

📦 Dependencies

List any dependencies or related PRs (e.g., "Depends on #123").

🐛 Related Issues

Link any issues that are resolved or affected by this PR.

✅ Contributor Checklist

I have tested these changes with kustomize. See Installation Prerequisites.
All commits are signed-off to satisfy the DCO check.
I have considered adding my company to the adopters page to support Kubeflow and help the community, since I expect help from the community for my issue (see 1. and 2.).

You can join the CNCF Slack and access our meetings at the Kubeflow Community website. Our channel on the CNCF Slack is here #kubeflow-platform.

Signed-off-by: kunal-511 <yoyokvunal@gmail.com>

google-oss-prow · 2025-07-01T18:12:49Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign kimwnasptd for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Signed-off-by: kunal-511 <yoyokvunal@gmail.com>

kunal-511 · 2025-07-01T19:59:00Z

tests/trainer_test.sh

+kubectl get pods -n $KF_PROFILE --show-labels
+kubectl wait --for=condition=Ready pod -l batch.kubernetes.io/job-name=pytorch-simple-node-0 -n $KF_PROFILE --timeout=180s
+
+kubectl wait --for=condition=Complete job/pytorch-simple-node-0 -n $KF_PROFILE --timeout=450s


@juliusvonkohout
this job is taking too much time to complete like around 25-30 minutes.
So should i change this condition to running or increase the timeout?

I would even like to lower the timeout if possible. 450 s is already a lot. I set only requests instead of limits now. that might help already. but if you can reduce it further, please do so.

Yes now all are green

Signed-off-by: Julius von Kohout <45896133+juliusvonkohout@users.noreply.github.com>

Signed-off-by: kunal-511 <yoyokvunal@gmail.com>

juliusvonkohout · 2025-07-03T08:25:35Z

tests/trainer_install.sh

+kubectl apply --server-side --force-conflicts -f overlays/kubeflow-platform/kubeflow-trainer-roles.yaml
+kustomize build overlays/runtimes | kubectl apply --server-side --force-conflicts -f -
+
+kubectl get deployment -n kubeflow-system kubeflow-trainer-controller-manager


please switch everything to kubeflow instead of kubeflow-system for now. We need an overlay in applications/trainer/overlays/... for that. I would wait until we have other components ready before opening up a new namespaces. Also for that namespace kubeflow-system we would have to set PSS restricted before we use it and introduce it to users.

overlays/manager: Deploys to kubeflow-system namespace, Has a secretGenerator that creates the webhook certificate secret properly

We are using this overlays/kubeflow-platform: Changes the overall namespace to kubeflow and removes kubeflow-system namespace, but the secretGenerator still references kubeflow-system.

This creates a namespace mismatch for the webhook certificate which causing the test to fail

You could adjust the certificate. You can also go the other way around, so stick with kubeflow-system and add PSS to the kubeflow-system namespace and add default blocker networkpolicies for kubeflow-system and exceptions for the trainer webhook in kubeflow-system.

Signed-off-by: kunal-511 <yoyokvunal@gmail.com>

kunal-511 · 2025-07-15T14:50:49Z

/retest

google-oss-prow · 2025-07-15T14:51:04Z

@kunal-511: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

juliusvonkohout · 2025-07-15T15:53:14Z

/ok-to-test

Signed-off-by: kunal-511 <yoyokvunal@gmail.com>

juliusvonkohout · 2025-07-22T14:46:25Z

please fix the securitycontext here and upstream in the trainer repository.

Signed-off-by: kunal-511 <yoyokvunal@gmail.com>

kunal-511 · 2025-07-22T18:24:16Z

@juliusvonkohout I think this ready for the review

kunal-511 · 2025-07-22T18:36:29Z

/retest

kunal-511 · 2025-07-24T16:50:23Z

/retest

Signed-off-by: kunal-511 <yoyokvunal@gmail.com>

kunal-511 · 2025-07-25T15:49:50Z

/retest

Signed-off-by: kunal-511 <yoyokvunal@gmail.com>

juliusvonkohout · 2025-07-28T10:26:26Z

experimental/security/PSS/jobset-pss-fix.yaml

lets not have this under experimental, this is really a basic requirement now. It can be directly in the trainer folder.

and please create an upstream PR to fix this and ping @andreyvelich to merge it.

juliusvonkohout · 2025-07-28T10:26:29Z

experimental/security/PSS/trainer-pss-fix.yaml

lets not have this under experimental, this is really a basic requirement now. It can be directly in the trainer folder.

and please create an upstream PR to fix this and ping @andreyvelich to merge it.

kunal-511 added 2 commits July 1, 2025 23:39

Update kubeflow/trainer manifests from master

ff92bfc

Signed-off-by: kunal-511 <yoyokvunal@gmail.com>

Sync Script Trainer2.0

93c908e

Signed-off-by: kunal-511 <yoyokvunal@gmail.com>

google-oss-prow bot requested review from juliusvonkohout and kimwnasptd July 1, 2025 18:12

google-oss-prow bot added the size/XXL label Jul 1, 2025

kunal-511 added 5 commits July 2, 2025 00:10

tests for trainer

f7b9c8d

Signed-off-by: kunal-511 <yoyokvunal@gmail.com>

Added pause to avoid conflict

684e92b

Signed-off-by: kunal-511 <yoyokvunal@gmail.com>

updated to trainjobs

c0937f1

Signed-off-by: kunal-511 <yoyokvunal@gmail.com>

Added trainer to end to end test

8d884e3

Signed-off-by: kunal-511 <yoyokvunal@gmail.com>

Fixed lint

2852ad7

Signed-off-by: kunal-511 <yoyokvunal@gmail.com>

kunal-511 commented Jul 1, 2025

View reviewed changes

juliusvonkohout and others added 2 commits July 2, 2025 11:43

Update trainer_job.yaml

fff525d

Signed-off-by: Julius von Kohout <45896133+juliusvonkohout@users.noreply.github.com>

Reduced the timeout

df7a1d4

Signed-off-by: kunal-511 <yoyokvunal@gmail.com>

juliusvonkohout reviewed Jul 3, 2025

View reviewed changes

kunal-511 added 2 commits July 3, 2025 14:43

Updated namespace to kubeflow

33d9cbc

Signed-off-by: kunal-511 <yoyokvunal@gmail.com>

changed to overlays/kubeflow-platform

9fd6196

Signed-off-by: kunal-511 <yoyokvunal@gmail.com>

kunal-511 mentioned this pull request Jul 5, 2025

Synchronize training operator manifests v2 #3179

Closed

3 tasks

google-oss-prow bot added the ok-to-test label Jul 15, 2025

kunal-511 added 8 commits July 17, 2025 00:43

changed to kubeflow-system namespace and add proper security policies

bde824e

Signed-off-by: kunal-511 <yoyokvunal@gmail.com>

lint fix

2a33034

Signed-off-by: kunal-511 <yoyokvunal@gmail.com>

added wait for trainjob

2178090

Signed-off-by: kunal-511 <yoyokvunal@gmail.com>

Updated wait

b4d12fd

Signed-off-by: kunal-511 <yoyokvunal@gmail.com>

removed wait

c1f6458

Signed-off-by: kunal-511 <yoyokvunal@gmail.com>

added sleep

e6e2505

Signed-off-by: kunal-511 <yoyokvunal@gmail.com>

added pod ready condition

f233351

Signed-off-by: kunal-511 <yoyokvunal@gmail.com>

Updated the command

92e96a1

Signed-off-by: kunal-511 <yoyokvunal@gmail.com>

kunal-511 added 4 commits July 22, 2025 17:00

Add more logs

d10c4dd

Signed-off-by: kunal-511 <yoyokvunal@gmail.com>

checking jobset webhook pod readiness

5f49e5b

Signed-off-by: kunal-511 <yoyokvunal@gmail.com>

network connectivity tests

46ea825

Signed-off-by: kunal-511 <yoyokvunal@gmail.com>

add:jobset-webhook-kubeflow-system

9fc538c

Signed-off-by: kunal-511 <yoyokvunal@gmail.com>

kunal-511 marked this pull request as draft July 22, 2025 12:00

google-oss-prow bot added the do-not-merge/work-in-progress label Jul 22, 2025

kunal-511 added 4 commits July 22, 2025 17:42

Force restart of Jobset controller

22f1498

Signed-off-by: kunal-511 <yoyokvunal@gmail.com>

allow for all

a28be92

Signed-off-by: kunal-511 <yoyokvunal@gmail.com>

allow for all

9a33188

Signed-off-by: kunal-511 <yoyokvunal@gmail.com>

clean logs

bb35492

Signed-off-by: kunal-511 <yoyokvunal@gmail.com>

kunal-511 marked this pull request as ready for review July 22, 2025 12:58

google-oss-prow bot removed the do-not-merge/work-in-progress label Jul 22, 2025

google-oss-prow bot requested review from juliusvonkohout and TobiasGoerke July 22, 2025 12:59

kunal-511 added 3 commits July 22, 2025 18:36

clean logs

286413a

Signed-off-by: kunal-511 <yoyokvunal@gmail.com>

Fixed lint

9c7b01b

Signed-off-by: kunal-511 <yoyokvunal@gmail.com>

clean logs

db3c68c

Signed-off-by: kunal-511 <yoyokvunal@gmail.com>

kunal-511 added 2 commits July 22, 2025 23:39

patch added

1e2fa13

Signed-off-by: kunal-511 <yoyokvunal@gmail.com>

add patch

c8b4db7

Signed-off-by: kunal-511 <yoyokvunal@gmail.com>

Removed the restart

6014d85

Signed-off-by: kunal-511 <yoyokvunal@gmail.com>

kunal-511 added 3 commits July 25, 2025 21:26

Removed the restart

48f2615

Signed-off-by: kunal-511 <yoyokvunal@gmail.com>

Add restart

98a9578

Signed-off-by: kunal-511 <yoyokvunal@gmail.com>

added wait for restart

be02c58

Signed-off-by: kunal-511 <yoyokvunal@gmail.com>

juliusvonkohout reviewed Jul 28, 2025

View reviewed changes

Synchronize trainer manifests 2.0 #3181

Are you sure you want to change the base?

Synchronize trainer manifests 2.0 #3181

Uh oh!

Conversation

kunal-511 commented Jul 1, 2025

Pull Request Template for Kubeflow Manifests

✏️ Summary of Changes

📦 Dependencies

🐛 Related Issues

✅ Contributor Checklist

Uh oh!

google-oss-prow bot commented Jul 1, 2025

Uh oh!

kunal-511 Jul 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

juliusvonkohout Jul 2, 2025

Choose a reason for hiding this comment

Uh oh!

kunal-511 Jul 2, 2025

Choose a reason for hiding this comment

Uh oh!

juliusvonkohout Jul 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kunal-511 Jul 16, 2025

Choose a reason for hiding this comment

Uh oh!

juliusvonkohout Jul 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kunal-511 commented Jul 15, 2025

Uh oh!

google-oss-prow bot commented Jul 15, 2025

Uh oh!

juliusvonkohout commented Jul 15, 2025

Uh oh!

juliusvonkohout commented Jul 22, 2025

Uh oh!

kunal-511 commented Jul 22, 2025

Uh oh!

kunal-511 commented Jul 22, 2025

Uh oh!

kunal-511 commented Jul 24, 2025

Uh oh!

kunal-511 commented Jul 25, 2025

Uh oh!

juliusvonkohout Jul 28, 2025

Choose a reason for hiding this comment

Uh oh!

juliusvonkohout Jul 28, 2025

Choose a reason for hiding this comment

Uh oh!

juliusvonkohout Jul 28, 2025

Choose a reason for hiding this comment

Uh oh!

juliusvonkohout Jul 28, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kunal-511 Jul 1, 2025 •

edited

Loading

juliusvonkohout Jul 3, 2025 •

edited

Loading

juliusvonkohout Jul 16, 2025 •

edited

Loading