Upgrade Kubernetes Version to 1.32.2 #2521

Electronic-Waste · 2025-03-02T05:59:35Z

What this PR does / why we need it:

We should support the latest k8s v1.32 release https://github.com/kubernetes/kubernetes/releases/tag/v1.32.2.

Kubeflow Trainer has decided to support Kubernetes v1.32 kubeflow/trainer#2434. We'd better upgrade the Kubernetes version to maintain consistency with Trainer.

/cc @kubeflow/wg-automl-leads @helenxie-bit @mahdikhashan @Doris-xm @truc0

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #2516

Checklist:

Docs included if any changes are user facing

google-oss-prow · 2025-03-02T05:59:43Z

@Electronic-Waste: GitHub didn't allow me to request PR reviews from the following users: truc0.

Note that only kubeflow members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

What this PR does / why we need it:

We should support the latest k8s v1.32 release https://github.com/kubernetes/kubernetes/releases/tag/v1.32.2.

Kubeflow Trainer has decided to support Kubernetes v1.32 kubeflow/trainer#2434. We'd better upgrade the Kubernetes version to maintain consistency with Trainer.

/cc @kubeflow/wg-automl-leads @helenxie-bit @mahdikhashan @Doris-xm @truc0

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #2516

Checklist:

Docs included if any changes are user facing

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Electronic-Waste

This PR is ready for the review. PTAL if you have time👀

@kubeflow/wg-automl-leads @helenxie-bit @mahdikhashan @Doris-xm @truc0

Electronic-Waste · 2025-03-02T12:01:51Z

pkg/controller.v1beta1/trial/trial_controller_test.go

+	succeededBatchJobName = "test-job-succeeded"
+	failedBatchJobName    = "test-job-failed"


We need to separate the Trial run with "Failed" BatchJob. test and other tests. That's because:

If a Job is succeeded, we need to set Complete and SuccessCriteriaMet Conditions in the Job status

If a Job is failed, we need to set Failed and FailureTarget Conditions in the Job status

And we can't mutate these completion conditions, or we will get:

Invalid value: cannot disable the terminal Failed=True condition, status.conditions: Invalid value: cannot disable the terminal FailureTarget=True condition,

So we need to separate these UTs with two different Job Names.

Electronic-Waste · 2025-03-02T12:05:54Z

pkg/controller.v1beta1/trial/trial_controller_test.go

+			// TODO(Electronic-Waste): Remove this condition when K8s 1.30 is no longer supported.
+			// SuccessPolicy is available in K8s 1.31 and later. If we set it in K8s 1.30, it will be ignored.
+			// And when we set the status with `SuccessCriteriaMet`, it will report error:
+			// "Invalid value: cannot set SuccessCriteriaMet=True for Job without SuccessPolicy".
+			// REF: https://kubernetes.io/docs/concepts/workloads/controllers/job/#success-policy
+			isK8sVersion130 := strings.Contains(os.Getenv("KUBEBUILDER_ASSETS"), "1.30")


We need to remove this when we do not plan to support Kubernetes v1.30.

Related CI error: https://github.com/kubeflow/katib/actions/runs/13613616097/job/38053846460

Electronic-Waste · 2025-03-03T02:32:33Z

/rerun-all

Electronic-Waste · 2025-03-03T04:00:18Z

Makefile

 lint:
 ifndef HAS_LINT
-	go install github.com/golangci/golangci-lint/cmd/golangci-lint@v1.57.2
+	go install github.com/golangci/golangci-lint/cmd/golangci-lint@v1.60.1


Update this because support for Go 1.23 starts from v1.60.1

https://golangci-lint.run/product/changelog/#v1601

Electronic-Waste · 2025-03-03T07:57:34Z

/rerun-all

github-actions · 2025-06-05T05:07:02Z

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

Electronic-Waste · 2025-06-05T07:15:10Z

/remove-lifecycle stale

mahdikhashan · 2025-06-05T09:31:29Z

.github/workflows/test-go.yaml

      matrix:
        # Detail: `setup-envtest list`
-        kubernetes-version: ["1.29.3", "1.30.0", "1.31.0"]
+        kubernetes-version: ["1.29.3", "1.30.0", "1.31.0", "1.32.0"]


do we need to have 1.32.2 here as well?

Sorry for the late response. If I remember correctly, that's because envtest did not have 1.32.2 version.

mahdikhashan · 2025-06-05T09:33:26Z

SGTM, (i dono why / l g t m didn't work)

github-actions · 2025-09-03T10:06:28Z

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

andreyvelich · 2025-09-03T12:59:49Z

/remove-lifecycle stale
@Electronic-Waste Could you finalize this PR please ?
Maybe we can directly upgrade to k8s 1.34 like in Trainer: kubeflow/trainer#2804

Electronic-Waste · 2025-09-05T06:22:21Z

@Electronic-Waste Could you finalize this PR please ?

Sure. I'll resolve the conflict.

Maybe we can directly upgrade to k8s 1.34 like in Trainer: kubeflow/trainer#2804

I don't think it's a good practice to directly upgrade k8s version to 1.34. We need to gradually upgrade the version so that we can roll back to the older version if faced with incompatibility:)

WDYT? @andreyvelich

…elease-0.20. Signed-off-by: Electronic-Waste <2690692950@qq.com>

Signed-off-by: Electronic-Waste <2690692950@qq.com>

andreyvelich · 2025-09-05T10:10:25Z

I don't think it's a good practice to directly upgrade k8s version to 1.34. We need to gradually upgrade the version so that we can roll back to the older version if faced with incompatibility:)

Sure, we can split it between 2 PRs, but in the next Katib release I want us to support same k8s version as in Trainer.

Electronic-Waste · 2025-09-05T12:31:23Z

@andreyvelich Sure, I'll create another PR after this

Electronic-Waste · 2025-09-05T12:31:30Z

/retest

andreyvelich

@Electronic-Waste Are we ready to merge this ?

Electronic-Waste · 2025-09-13T04:50:51Z

@andreyvelich I think we're ready to merge this.

andreyvelich

/lgtm
/approve
@Electronic-Waste Please create the followup PR to support k8s 1.33

google-oss-prow · 2025-09-13T15:19:15Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: andreyvelich

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [andreyvelich]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

google-oss-prow bot requested review from a team, Doris-xm, helenxie-bit and mahdikhashan March 2, 2025 05:59

google-oss-prow bot added the size/XXL label Mar 2, 2025

Electronic-Waste force-pushed the feat/upgrade-k8s-1.32 branch from b7eea92 to 69230b5 Compare March 2, 2025 08:15

Electronic-Waste marked this pull request as draft March 2, 2025 08:21

google-oss-prow bot added the do-not-merge/work-in-progress label Mar 2, 2025

Electronic-Waste commented Mar 2, 2025

View reviewed changes

Electronic-Waste marked this pull request as ready for review March 2, 2025 12:06

google-oss-prow bot removed the do-not-merge/work-in-progress label Mar 2, 2025

Electronic-Waste commented Mar 3, 2025

View reviewed changes

Electronic-Waste mentioned this pull request Mar 14, 2025

chore: Upgrade Go version to 1.23 #2526

Merged

1 task

github-actions bot added the lifecycle/stale label Jun 5, 2025

google-oss-prow bot removed the lifecycle/stale label Jun 5, 2025

mahdikhashan reviewed Jun 5, 2025

View reviewed changes

github-actions bot added the lifecycle/stale label Sep 3, 2025

google-oss-prow bot removed the lifecycle/stale label Sep 3, 2025

Electronic-Waste added 3 commits September 5, 2025 06:41

chore(Makefile): upgrade envtest version to 1.32 & setup-envtest to r…

126e91d

…elease-0.20. Signed-off-by: Electronic-Waste <2690692950@qq.com>

chore(gomod): upgrade go version to 1.23.0 & update package version.

f0f33d8

Signed-off-by: Electronic-Waste <2690692950@qq.com>

chore: make generate.

1c44031

Signed-off-by: Electronic-Waste <2690692950@qq.com>

Electronic-Waste added 16 commits September 5, 2025 06:43

fix(ci): add 1.32 support & remove 1.29 support.

cbe2c87

Signed-off-by: Electronic-Waste <2690692950@qq.com>

fix(test): add FailureTarget condition to job & add startTime.

7a423b9

Signed-off-by: Electronic-Waste <2690692950@qq.com>

fix(test): add SuccessCriteriaMet condition.

f05a9da

Signed-off-by: Electronic-Waste <2690692950@qq.com>

fix(test): add StartTime.

c28146a

Signed-off-by: Electronic-Waste <2690692950@qq.com>

fix(test): create a new Job after the failed job.

f9a0422

Signed-off-by: Electronic-Waste <2690692950@qq.com>

fix(test): move the failed job test to the end.

ffa9c8e

Signed-off-by: Electronic-Waste <2690692950@qq.com>

fix(test): update job to indexed job.

13a1219

Signed-off-by: Electronic-Waste <2690692950@qq.com>

fix(test): seperate job name between failed job and succeeded job.

6be3387

Signed-off-by: Electronic-Waste <2690692950@qq.com>

fix(test): remove SuccessPolicy.

33403ef

Signed-off-by: Electronic-Waste <2690692950@qq.com>

fix(test): disable SuccessCriteriaMet in 1.30.

bb0f6d6

Signed-off-by: Electronic-Waste <2690692950@qq.com>

fix(test): delete some fields in Job.

d5443c4

Signed-off-by: Electronic-Waste <2690692950@qq.com>

fix(test): update comments.

16e4fb6

Signed-off-by: Electronic-Waste <2690692950@qq.com>

fix(test): update comments.

dc1d87a

Signed-off-by: Electronic-Waste <2690692950@qq.com>

fix(ci): readd v1.29 version.

c317024

Signed-off-by: Electronic-Waste <2690692950@qq.com>

fix(test): fix error in ci.

d169b01

Signed-off-by: Electronic-Waste <2690692950@qq.com>

fix(test): fix version error.

0fb2f28

Signed-off-by: Electronic-Waste <2690692950@qq.com>

Electronic-Waste force-pushed the feat/upgrade-k8s-1.32 branch from ee27a16 to 0fb2f28 Compare September 5, 2025 06:45

andreyvelich reviewed Sep 13, 2025

View reviewed changes

google-oss-prow bot assigned andreyvelich Sep 13, 2025

google-oss-prow bot added the lgtm label Sep 13, 2025

google-oss-prow bot added the approved label Sep 13, 2025

google-oss-prow bot merged commit 82260cb into kubeflow:master Sep 13, 2025
83 of 84 checks passed

		succeededBatchJobName = "test-job-succeeded"
		failedBatchJobName = "test-job-failed"

Upgrade Kubernetes Version to 1.32.2 #2521

Upgrade Kubernetes Version to 1.32.2 #2521

Uh oh!

Conversation

Electronic-Waste commented Mar 2, 2025

Uh oh!

google-oss-prow bot commented Mar 2, 2025

Uh oh!

Electronic-Waste left a comment

Choose a reason for hiding this comment

Uh oh!

Electronic-Waste Mar 2, 2025

Choose a reason for hiding this comment

Uh oh!

Electronic-Waste Mar 2, 2025

Choose a reason for hiding this comment

Uh oh!

Electronic-Waste commented Mar 3, 2025

Uh oh!

Electronic-Waste Mar 3, 2025

Choose a reason for hiding this comment

Uh oh!

Electronic-Waste commented Mar 3, 2025

Uh oh!

github-actions bot commented Jun 5, 2025

Uh oh!

Electronic-Waste commented Jun 5, 2025

Uh oh!

mahdikhashan Jun 5, 2025

Choose a reason for hiding this comment

Uh oh!

Electronic-Waste Sep 13, 2025

Choose a reason for hiding this comment

Uh oh!

mahdikhashan commented Jun 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Sep 3, 2025

Uh oh!

andreyvelich commented Sep 3, 2025

Uh oh!

Electronic-Waste commented Sep 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

andreyvelich commented Sep 5, 2025

Uh oh!

Electronic-Waste commented Sep 5, 2025

Uh oh!

Electronic-Waste commented Sep 5, 2025

Uh oh!

andreyvelich left a comment

Choose a reason for hiding this comment

Uh oh!

Electronic-Waste commented Sep 13, 2025

Uh oh!

andreyvelich left a comment

Choose a reason for hiding this comment

Uh oh!

google-oss-prow bot commented Sep 13, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mahdikhashan commented Jun 5, 2025 •

edited

Loading

Electronic-Waste commented Sep 5, 2025 •

edited

Loading