Update Manifest Images to GHCR #2544

saileshd1402 · 2025-03-18T18:59:21Z

What this PR does / why we need it:
Update Manifest images to GHCR instead of Dockerhub. Required to decrease rate limiting issues

Which issue(s) this PR fixes (optional, in Fixes #<issue number>, #<issue number>, ... format, will close the issue(s) when PR gets merged):

Address this comment to update manifest images: #2491 (comment)

cc: @andreyvelich @Electronic-Waste @mahdikhashan @tenzen-y

Checklist:

Docs included if any changes are user facing

Signed-off-by: sailesh duddupudi <saileshradar@gmail.com>

andreyvelich

We might also need to update other places:
e.g. https://github.com/kubeflow/trainer/blob/release-1.9/pkg/config/config.go#L35

@kubeflow/wg-training-leads @Electronic-Waste What other places where we use our images ?

saileshd1402 · 2025-03-18T19:12:37Z

We might also need to update other places: e.g. https://github.com/kubeflow/trainer/blob/release-1.9/pkg/config/config.go#L35

Got it, I'll search and update

Signed-off-by: sailesh duddupudi <saileshradar@gmail.com>

review-notebook-app · 2025-03-18T19:43:34Z

Check out this pull request on

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

saileshd1402 · 2025-03-18T19:45:09Z

I've updated the images, PTAL

saileshd1402 · 2025-03-18T19:58:14Z

pkg/config/config.go

 	PyTorchInitContainerMaxTriesDefault = 100
 	// MPIKubectlDeliveryImageDefault is the default image for launcher pod in MPIJob init container.
-	MPIKubectlDeliveryImageDefault = "kubeflow/kubectl-delivery:latest"
+	MPIKubectlDeliveryImageDefault = "ghcr.io/kubeflow/training/kubectl-delivery:latest"


Since this change is in pkg folder, should we update image tag in overlay again in another PR to reflect this change?

We need to update tag in overlay after we make a release.
Here is the process: https://github.com/kubeflow/trainer/tree/release-1.9/docs/release#release-branches-and-tags

tenzen-y · 2025-03-19T02:37:53Z

Makefile

@@ -1,5 +1,5 @@
 # Image URL to use all building/pushing image targets
-IMG ?= kubeflow/training-operator:latest
+IMG ?= ghcr.io/kubeflow/training/training-operator:latest


Suggested change

IMG ?= ghcr.io/kubeflow/training/training-operator:latest

IMG ?= ghcr.io/kubeflow/trainer/training-operator:latest

Isn't training misstype?

I think it's an error that also happened in #2491.

Maybe we should also change the image name in:

trainer/.github/workflows/build-and-publish-images.yaml

Lines 76 to 78 in f654b1e

image: |

docker.io/kubeflow/${{ inputs.component-name }}

ghcr.io/kubeflow/training/${{ inputs.component-name }}

trainer/.github/workflows/build-and-publish-images.yaml

Lines 89 to 91 in f654b1e

image: |

docker.io/kubeflow/${{ inputs.component-name }}

ghcr.io/kubeflow/training/${{ inputs.component-name }}

WDYT? @saileshd1402 @andreyvelich

I think so too

@andreyvelich We may also need to delete the training/* image in the github package of trainer.

Fixed the error in: #2546

coveralls · 2025-03-19T02:41:15Z

coverage: 100.0%. remained the same
when pulling f6294f0 on saileshd1402:update-manifest-images-to-ghcr
into 5840e81 on kubeflow:release-1.9.

…est-images-to-ghcr

Signed-off-by: sailesh duddupudi <saileshradar@gmail.com>

Electronic-Waste

Thanks for this @saileshd1402. LGTM!

/lgtm

mahdikhashan · 2025-03-19T17:38:25Z

/lgtm

thank you for your time

saileshd1402 · 2025-03-19T19:17:49Z

@andreyvelich @Electronic-Waste do you think we can keep these changes or move to a different prefix? The other one discussed: /kubeflow/training-v1 also sounds good to me. I'm not in favour of /kubeflow/training-operator

andreyvelich · 2025-03-19T23:35:54Z

@saileshd1402 Let's keep the training-v1 prefix to be more precise.
Even that we have v1- in the tag of the images, I think having the prefix would be better for users.

Electronic-Waste · 2025-03-20T02:10:47Z

@saileshd1402 Let's keep the training-v1 prefix to be more precise.
Even that we have v1- in the tag of the images, I think having the prefix would be better for users.

+1

tenzen-y · 2025-03-20T06:55:57Z

@saileshd1402 Let's keep the training-v1 prefix to be more precise. Even that we have v1- in the tag of the images, I think having the prefix would be better for users.

SGTM

andreyvelich · 2025-03-20T12:37:05Z

@saileshd1402 Please could you update this PR with the /training-v1/ prefix, so I can cut the release ?

Signed-off-by: sailesh duddupudi <saileshradar@gmail.com>

saileshd1402 · 2025-03-20T14:20:59Z

manifests/overlays/standalone/kustomization.yaml

-  - name: kubeflow/training-operator
-    newTag: v1-5170a36
+  - name: ghcr.io/kubeflow/training-v1/training-operator
+    newTag: v1-f654b1e


will create a followup PR to fix this tag

andreyvelich · 2025-03-20T14:25:29Z

Thank you for the update @saileshd1402!
/lgtm
/assign @Electronic-Waste @tenzen-y @astefanutti @kubeflow/wg-training-leads @mahdikhashan

Electronic-Waste · 2025-03-21T03:26:19Z

Thanks for this @saileshd1402!

/lgtm

tenzen-y

Could you update

trainer/scripts/setup-training-operator.sh

Line 37 in 5840e81

kustomize edit set image kubeflow/training-operator=${REGISTRY}:${VERSION}

as well?

Signed-off-by: sailesh duddupudi <saileshradar@gmail.com>

google-oss-prow · 2025-03-21T09:07:20Z

New changes are detected. LGTM label has been removed.

google-oss-prow · 2025-03-21T09:07:24Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: Electronic-Waste
Once this PR has been reviewed and has the lgtm label, please ask for approval from andreyvelich. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

saileshd1402 · 2025-03-21T09:07:48Z

Could you update

trainer/scripts/setup-training-operator.sh

Line 37 in 5840e81

kustomize edit set image kubeflow/training-operator=${REGISTRY}:${VERSION}

as well?

Didn't notice that, thank you! updated

mahdikhashan · 2025-03-21T09:35:48Z

i wonder katib images should be called from ghcr?

examples/pytorch/simple.yaml:              image: docker.io/kubeflowkatib/pytorch-mnist:v1beta1-45c5727
examples/pytorch/simple.yaml:              image: docker.io/kubeflowkatib/pytorch-mnist:v1beta1-45c5727
pkg/webhooks/pytorch/pytorchjob_webhook_test.go:                                                Image:           "docker.io/kubeflowkatib/pytorch-mnist:v1beta1-45c5727",
pkg/webhooks/pytorch/pytorchjob_webhook_test.go:                                                Image:           "docker.io/kubeflowkatib/pytorch-mnist:v1beta1-45c5727",

ref: kubeflow/katib#2529

andreyvelich · 2025-03-21T11:16:25Z

Do we want to merge this PR without e2e pass or update the example images in the followup PR?

saileshd1402 · 2025-03-21T15:56:36Z

In the interest of time I would want to merge this. But either way works, please let me know

Update Manifest Images to GHCR

393a0be

Signed-off-by: sailesh duddupudi <saileshradar@gmail.com>

google-oss-prow bot requested review from jinchihe and kuizhiqing March 18, 2025 18:59

google-oss-prow bot added the size/XS label Mar 18, 2025

andreyvelich reviewed Mar 18, 2025

View reviewed changes

Update examples, SDK and pkg images to GHCR

9a2f500

Signed-off-by: sailesh duddupudi <saileshradar@gmail.com>

google-oss-prow bot added size/M and removed size/XS labels Mar 18, 2025

saileshd1402 commented Mar 18, 2025

View reviewed changes

tenzen-y reviewed Mar 19, 2025

View reviewed changes

Electronic-Waste mentioned this pull request Mar 19, 2025

fix(ci): Change publish dir from training to trainer #2546

Merged

1 task

saileshd1402 added 2 commits March 19, 2025 15:21

Merge remote-tracking branch 'upstream/release-1.9' into update-manif…

60be0c9

…est-images-to-ghcr

revert trainer to training in ghcr prefix

f6294f0

Signed-off-by: sailesh duddupudi <saileshradar@gmail.com>

Electronic-Waste approved these changes Mar 19, 2025

View reviewed changes

google-oss-prow bot assigned Electronic-Waste Mar 19, 2025

google-oss-prow bot added the lgtm label Mar 19, 2025

google-oss-prow bot assigned mahdikhashan Mar 19, 2025

update prefix

68838a8

Signed-off-by: sailesh duddupudi <saileshradar@gmail.com>

google-oss-prow bot removed the lgtm label Mar 20, 2025

saileshd1402 commented Mar 20, 2025

View reviewed changes

google-oss-prow bot assigned andreyvelich Mar 20, 2025

google-oss-prow bot added the lgtm label Mar 20, 2025

tenzen-y reviewed Mar 21, 2025

View reviewed changes

update image in setup script

b53fc32

Signed-off-by: sailesh duddupudi <saileshradar@gmail.com>

google-oss-prow bot removed the lgtm label Mar 21, 2025

andreyvelich merged commit 31da2d6 into kubeflow:release-1.9 Mar 24, 2025
34 of 53 checks passed

	IMG ?= ghcr.io/kubeflow/training/training-operator:latest
	IMG ?= ghcr.io/kubeflow/trainer/training-operator:latest

	image: \|
	docker.io/kubeflow/${{ inputs.component-name }}
	ghcr.io/kubeflow/training/${{ inputs.component-name }}

Update Manifest Images to GHCR #2544

Update Manifest Images to GHCR #2544

Uh oh!

Conversation

saileshd1402 commented Mar 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

andreyvelich left a comment

Choose a reason for hiding this comment

Uh oh!

saileshd1402 commented Mar 18, 2025

Uh oh!

review-notebook-app bot commented Mar 18, 2025

Uh oh!

saileshd1402 commented Mar 18, 2025

Uh oh!

saileshd1402 Mar 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andreyvelich Mar 18, 2025

Choose a reason for hiding this comment

Uh oh!

tenzen-y Mar 19, 2025

Choose a reason for hiding this comment

Uh oh!

Electronic-Waste Mar 19, 2025

Choose a reason for hiding this comment

Uh oh!

tenzen-y Mar 19, 2025

Choose a reason for hiding this comment

Uh oh!

Electronic-Waste Mar 19, 2025

Choose a reason for hiding this comment

Uh oh!

Electronic-Waste Mar 19, 2025

Choose a reason for hiding this comment

Uh oh!

coveralls commented Mar 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Electronic-Waste left a comment

Choose a reason for hiding this comment

Uh oh!

mahdikhashan commented Mar 19, 2025

Uh oh!

saileshd1402 commented Mar 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

andreyvelich commented Mar 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Electronic-Waste commented Mar 20, 2025

Uh oh!

tenzen-y commented Mar 20, 2025

Uh oh!

andreyvelich commented Mar 20, 2025

Uh oh!

saileshd1402 Mar 20, 2025

Choose a reason for hiding this comment

Uh oh!

andreyvelich commented Mar 20, 2025

Uh oh!

Electronic-Waste commented Mar 21, 2025

Uh oh!

tenzen-y left a comment

Choose a reason for hiding this comment

Uh oh!

google-oss-prow bot commented Mar 21, 2025

Uh oh!

google-oss-prow bot commented Mar 21, 2025

Uh oh!

saileshd1402 commented Mar 21, 2025

Uh oh!

mahdikhashan commented Mar 21, 2025

Uh oh!

andreyvelich commented Mar 21, 2025

Uh oh!

saileshd1402 commented Mar 21, 2025

Uh oh!

saileshd1402 commented Mar 18, 2025 •

edited

Loading

saileshd1402 Mar 18, 2025 •

edited

Loading

coveralls commented Mar 19, 2025 •

edited

Loading

saileshd1402 commented Mar 19, 2025 •

edited

Loading

andreyvelich commented Mar 19, 2025 •

edited

Loading