Upgrade to calico 3.29 and use windows support #5523

jsturtevant · 2025-03-27T22:33:03Z

What type of PR is this?

/kind cleanup
What this PR does / why we need it:
Upgrades Calico to 3.29 and updates windows to use the upstream Calico HPC support .

I tried to remove as much of the calico manifests but the Private Cluster test still uses those so they are still needed. We still need a bunch of the CSRs for windows since the calico still needs some additional support to fully remove it (tigera/operator#3113).

I did chat briefly with @Jont828 about expanding the helmProxy to take an additional field that would be Pre/Post yaml to deploy so we could potentially remove some of the extra yaml. But that is low priority and could be a follow up.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #5498

Special notes for your reviewer:

TODOs:

squashed commits
includes documentation
adds unit tests
cherry-pick candidate

Release note:

Update Calico to 3.29

k8s-ci-robot · 2025-03-27T22:33:12Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign fabriziopandini for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

codecov · 2025-03-27T22:39:03Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 53.26%. Comparing base (09830c0) to head (87199fa).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5523      +/-   ##
==========================================
- Coverage   53.26%   53.26%   -0.01%     
==========================================
  Files         272      272              
  Lines       29529    29529              
==========================================
- Hits        15730    15728       -2     
- Misses      12984    12986       +2     
  Partials      815      815

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

templates/addons/calico/values.yaml

jsturtevant · 2025-03-28T23:45:51Z

leaving my self some notes after investigating the failures:

When kubernetes-services-endpoint is included the operator fails to install things, which can be solved with dns changes: If "KUBERNETES_SERVICE_HOST" is domain, initial installation is not possible. (tigera-operator for windows) projectcalico/calico#9536
the service cidrs are not being set properly in the helm template:
- values: "installation:\n cni:\n type: Calico\n calicoNetwork:\n bgp: Disabled\n \ windowsDataplane: HNS\n mtu: 1350\n ipPools:\n ipPools:\n - cidr: 192.168.0.0/16\n encapsulation: VXLAN\n serviceCIDRs: \n registry: mcr.microsoft.com/oss\n# Image and registry configuration for the tigera/operator pod.\ntigeraOperator:\n \ image: tigera/operator\n registry: mcr.microsoft.com/oss\ncalicoctl:\n image: mcr.microsoft.com/oss/calico/ctl\n"

jsturtevant · 2025-03-28T23:51:45Z

When kubernetes-services-endpoint is included the operator fails to install things, which can be solved with dns changes: If "KUBERNETES_SERVICE_HOST" is domain, initial installation is not possible. (tigera-operator for windows) projectcalico/calico#9536

not sure what to do about this one as the chart doesn't let me override dnsConfig settings

jsturtevant · 2025-03-31T23:40:14Z

with the latest changes the cluster is coming online:

Cluster capz-j9dp7i created and fully operational
NAME                              STATUS   ROLES           AGE     VERSION    INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                         KERNEL-VERSION     CONTAINER-RUNTIME
capz-j9dp-55m7p                   Ready    <none>          4m7s    v1.29.10   10.1.0.4      <none>        Windows Server 2022 Datacenter   10.0.20348.2762    containerd://1.7.20
capz-j9dp-dljnd                   Ready    <none>          4m5s    v1.29.10   10.1.0.6      <none>        Windows Server 2022 Datacenter   10.0.20348.2762    containerd://1.7.20
capz-j9dp7i-control-plane-hgrj5   Ready    control-plane   6m40s   v1.29.10   10.0.0.4      <none>        Ubuntu 24.04.1 LTS               6.8.0-1016-azure   containerd://1.7.20
capz-j9dp7i-md-0-pb8j6-xnkct      Ready    <none>          5m1s    v1.29.10   10.1.0.5      <none>        Ubuntu 24.04.1 LTS               6.8.0-1016-azure   containerd://1.7.20
capz-j9dp7i-md-0-pb8j6-zc2cm      Ready    <none>          4m54s   v1.29.10   10.1.0.7      <none>        Ubuntu 24.04.1 LTS               6.8.0-1016-azure   containerd://1.7.20

jsturtevant · 2025-04-01T13:52:54Z

When kubernetes-services-endpoint is included the operator fails to install things, which can be solved with dns changes: If "KUBERNETES_SERVICE_HOST" is domain, initial installation is not possible. (tigera-operator for windows) projectcalico/calico#9536

not sure what to do about this one as the chart doesn't let me override dnsConfig settings

Opened projectcalico/calico#10117

jsturtevant · 2025-04-01T13:59:05Z

The workload upgrade test is failing with cloud manager crashing:

E0401 00:45:09.324689       1 runtime.go:79] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
goroutine 321 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic({0x20b7500, 0x3b02f10})
	/go/src/sigs.k8s.io/cloud-provider-azure/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:75 +0x85
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc000783a40?})
	/go/src/sigs.k8s.io/cloud-provider-azure/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:49 +0x6b
panic({0x20b7500?, 0x3b02f10?})
	/usr/local/go/src/runtime/panic.go:785 +0x132
sigs.k8s.io/cloud-provider-azure/pkg/provider.(*Cloud).getRouteTable(0x485d95?, 0x483d17?)
	/go/src/sigs.k8s.io/cloud-provider-azure/pkg/provider/azure_vmsets_repo.go:183 +0x4f

Is this a known issue? I don't see anything that makes it look like this is related to changes here.

jsturtevant · 2025-04-01T22:03:33Z

This is now mostly passing, It requires:

a release of the upstream Calico helm chart but the change as been checked in.
Images for the Windows Calico in the same location as the linux ones (working internally to solve this issue)

nawazkh · 2025-04-02T16:38:46Z

Is this a known issue? I don't see anything that makes it look like this is related to changes here.

Super strange that it is flaky with a panic..

jsturtevant · 2025-04-11T16:59:58Z

All the major challenges have been addresses. Just waiting on a Calico release that contains the updated helm chart

nawazkh · 2025-05-08T16:30:55Z

PR needs rebase.

We probably need a rebase here ?

jsturtevant · 2025-05-13T13:56:50Z

Still waiting on a helm chart release. I saw they released the next calico version but don't see the updates in the helm chart so I will ping them to find out if that will be released too

jsturtevant · 2025-05-13T19:59:28Z

Still waiting on a helm chart release. I saw they released the next calico version but don't see the updates in the helm chart so I will ping them to find out if that will be released too

It wasn't scheduled till Calico 3.31 but I requested some backports so should have it in the next patch release! projectcalico/calico#10117 (comment)

Signed-off-by: James Sturtevant <jsturtevant@gmail.com>

willie-yao

Looks mostly ready to me! Just some minor comments and questions from my end

willie-yao · 2025-05-27T21:28:40Z

config/capz/manager_image_patch.yaml

@@ -8,5 +8,5 @@ spec:
    spec:
      containers:
      # Change the value of image field below to your controller image URL
-      - image: gcr.io/k8s-staging-cluster-api-azure/cluster-api-azure-controller:main
+      - image: jjsacrpub.azurecr.io/cluster-api-azure/cluster-api-azure-controller-amd64:dev


Was this meant to be committed?

no, I think is result of running tools locally. Not sure why its not a part of the .gitignore? will remove

willie-yao · 2025-05-27T21:28:53Z

config/capz/manager_pull_policy.yaml

@@ -8,4 +8,4 @@ spec:
    spec:
      containers:
      - name: manager
-        imagePullPolicy: Always
+        imagePullPolicy: IfNotPresent


Same as above. I think these get changed when you run make tilt-up

yes, I think so... I'll remove it and squash commits too

willie-yao · 2025-05-27T21:35:31Z

templates/addons/calico.yaml

@@ -1,5 +1,1088 @@
 apiVersion: apiextensions.k8s.io/v1
 kind: CustomResourceDefinition
+metadata:


Are these diffs just generated templates from the new Calico source? And same with all the deletions in the other folders of addons?

Yes, this is automated via the make file command where it grabs the latest release then extracts it.

willie-yao · 2025-05-27T21:37:08Z

templates/addons/windows/calico/calico.yaml

-    plural: ""
-  conditions: []
-  storedVersions: []
+---


nit: eof line

willie-yao · 2025-05-27T21:39:21Z

test/e2e/data/infrastructure-azure/v1beta1/cluster-template.yaml

+    api server name properly\n# https://github.com/projectcalico/calico/issues/9536\ndnsConfig:
+    \n  nameservers:\n    - 127.0.0.53\n  options:\n    - name: edns0\n    - name:
+    trust-ad\nkubernetesServiceEndpoint:\n  host: \"{{ .Cluster.spec.controlPlaneEndpoint.host
+    }}\"\n  port: \"{{ .Cluster.spec.controlPlaneEndpoint.port }}\""


Can this be formatted like the previous diff so it's a bit more readable rather than using newline characters?

I can take a look, I think this is an automatically generated file?

github-project-automation bot added this to CAPZ Planning Mar 27, 2025

github-project-automation bot moved this to Todo in CAPZ Planning Mar 27, 2025

k8s-ci-robot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Mar 27, 2025

k8s-ci-robot requested review from marosset and willie-yao March 27, 2025 22:33

jsturtevant mentioned this pull request Mar 27, 2025

Update Calico to 3.29 #5499

Closed

4 tasks

k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels Mar 27, 2025

jsturtevant commented Mar 27, 2025

View reviewed changes

templates/addons/calico/values.yaml Show resolved Hide resolved

jsturtevant force-pushed the upgrade-calico-3.29 branch from 5fbed9d to 78297a9 Compare March 31, 2025 22:04

jsturtevant mentioned this pull request Apr 1, 2025

Add support for custom DNS in the operator projectcalico/calico#10117

Merged

3 tasks

jsturtevant mentioned this pull request Apr 1, 2025

Update all the kubernetes versions to 1.32 #5528

Open

4 tasks

jsturtevant mentioned this pull request Apr 2, 2025

Propose Windows operator updates tigera/operator#3113

Open

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 8, 2025

jsturtevant mentioned this pull request Apr 11, 2025

Remove calico in favor of upstream support kubernetes-sigs/sig-windows-tools#383

Open

3 tasks

jsturtevant force-pushed the upgrade-calico-3.29 branch from 4285b61 to 1f6f8b4 Compare April 30, 2025 20:36

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 30, 2025

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 3, 2025

This was referenced May 13, 2025

[release-v3.30] Auto pick #10117: Add support for custom DNS in the operator projectcalico/calico#10406

Merged

[release-v3.29] Auto pick #10117: Add support for custom DNS in the operator projectcalico/calico#10407

Merged

jsturtevant force-pushed the upgrade-calico-3.29 branch from 1f6f8b4 to 06b1720 Compare May 23, 2025 20:28

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 23, 2025

jsturtevant added 16 commits May 23, 2025 15:34

Upgrade to calico 3.29 and use windows support

26f0f7b

Signed-off-by: James Sturtevant <jsturtevant@gmail.com>

Remove the copy of the kubeadm config

a927ae9

Signed-off-by: James Sturtevant <jsturtevant@gmail.com>

testing if this is whats causing failures in calico controller

ac94864

Signed-off-by: James Sturtevant <jsturtevant@gmail.com>

remove unused

9f43255

Signed-off-by: James Sturtevant <jsturtevant@gmail.com>

regenerate templates

d5362d3

Signed-off-by: James Sturtevant <jsturtevant@gmail.com>

include dash?

fd2ab4a

Signed-off-by: James Sturtevant <jsturtevant@gmail.com>

hard code the service cidrs

868a9e3

Signed-off-by: James Sturtevant <jsturtevant@gmail.com>

Add the service endpoint back in

621a120

Signed-off-by: James Sturtevant <jsturtevant@gmail.com>

actually update the files

ddd89d3

Signed-off-by: James Sturtevant <jsturtevant@gmail.com>

Use a temp chart fix

0f53679

Signed-off-by: James Sturtevant <jsturtevant@gmail.com>

fix custom registry name

f2cffca

Signed-off-by: James Sturtevant <jsturtevant@gmail.com>

Don't need to know if cni is windows anymore

c0c24ff

Signed-off-by: James Sturtevant <jsturtevant@gmail.com>

ran generation after rebase

cb0c7ec

Signed-off-by: James Sturtevant <jsturtevant@gmail.com>

Update to the latest upstream version

9196107

Signed-off-by: James Sturtevant <jsturtevant@gmail.com>

Don't use images from MCR

44d1cd1

Signed-off-by: James Sturtevant <jsturtevant@gmail.com>

re-run generate

87199fa

Signed-off-by: James Sturtevant <jsturtevant@gmail.com>

jsturtevant force-pushed the upgrade-calico-3.29 branch from 47b9852 to 87199fa Compare May 23, 2025 22:37

willie-yao reviewed May 27, 2025

View reviewed changes

Upgrade to calico 3.29 and use windows support #5523

Are you sure you want to change the base?

Upgrade to calico 3.29 and use windows support #5523

Uh oh!

Conversation

jsturtevant commented Mar 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

k8s-ci-robot commented Mar 27, 2025

Uh oh!

codecov bot commented Mar 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

jsturtevant commented Mar 28, 2025

Uh oh!

jsturtevant commented Mar 28, 2025

Uh oh!

jsturtevant commented Mar 31, 2025

Uh oh!

jsturtevant commented Apr 1, 2025

Uh oh!

jsturtevant commented Apr 1, 2025

Uh oh!

jsturtevant commented Apr 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nawazkh commented Apr 2, 2025

Uh oh!

jsturtevant commented Apr 11, 2025

Uh oh!

nawazkh commented May 8, 2025

Uh oh!

jsturtevant commented May 13, 2025

Uh oh!

jsturtevant commented May 13, 2025

Uh oh!

willie-yao left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jsturtevant commented Mar 27, 2025 •

edited

Loading

codecov bot commented Mar 27, 2025 •

edited

Loading

jsturtevant commented Apr 1, 2025 •

edited

Loading