Add topologySpreadConstraints configuration to pod spec. #2530

laiminhtrung1997 · 2024-02-04T15:34:46Z

Dear all,

I think we should configure topologySpreadConstraints to pod spec so these pods can spread zones for high availability.

Could someone review it, please? Thank you very much.

Best regards.

monotek · 2024-05-16T11:22:19Z

We need that feature too.

FxKu · 2024-06-26T13:39:50Z

pkg/cluster/cluster.go

 	}
+	if !reflect.DeepEqual(c.Statefulset.Spec.Template.Spec.TopologySpreadConstraints, statefulSet.Spec.Template.Spec.TopologySpreadConstraints) {
+		needsReplace = true
+		needsRollUpdate = true


does this really need to trigger a rolling update of pods executed by operator? Will not K8s take care of it then once the statefulset is replaced?

I see, but is this wrong too?
https://github.com/zalando/postgres-operator/blob/master/pkg/cluster/cluster.go#L472

hm good point. Maybe we can leave as is for now. With rolling update we make sure pods immediately adhere the new constraints.

FxKu · 2024-06-26T13:43:49Z

pkg/apis/acid.zalan.do/v1/postgresql_type.go

 	InitContainersOld       []v1.Container `json:"init_containers,omitempty"`
 	PodPriorityClassNameOld string         `json:"pod_priority_class_name,omitempty"`
+
+	TopologySpreadConstraints []v1.TopologySpreadConstraint `json:"topologySpreadConstraints,omitempty"`


when introducing new fields to the api you have to update the CRDs, too. Unfortunately, we don't use a modern framework where this can be generated and you have to the manifests and in go code.

Plus, you need to run code generation -> ./hack/update-codegen.sh

FxKu · 2024-06-26T13:45:26Z

pkg/cluster/k8sres.go

 		false,
 		"",
 		false,
+		[]v1.TopologySpreadConstraint{},


atm we reuse configured tolerations also for logical backup so I guess we can do the same with constraints

Ok I got it.

FxKu · 2024-06-26T13:46:17Z

pkg/cluster/k8sres.go

 	return podAntiAffinity
 }

+func generateTopologySpreadConstraints(labels labels.Set, topologySpreadConstraintObjs []v1.TopologySpreadConstraint) []v1.TopologySpreadConstraint {


would like to see a unit test for this function :)

I added a unit test for this function.

FxKu · 2024-06-26T13:47:40Z

Can you also write an e2e test that tests that the constraints work as expected, please?

laiminhtrung1997 · 2024-06-27T08:34:56Z

Can you also write an e2e test that tests that the constraints work as expected, please?

Dear @FxKu
Thanks so much for your comments, I haven't written the UT or e2e test many, but I'll try my best. I'll mark it ready and let you know when I fixed the comments.
Best regards.

FxKu · 2024-11-14T11:35:09Z

Advice for the future: Don't force push and squash your commits in the middle of a review. Now it's super hard for me to see what feedback you've reflected and I have to review everything again 😞

laiminhtrung1997 · 2024-11-15T02:12:46Z

@FxKu

It is note. I am truly sorry for this. It will not happen again.

laiminhtrung1997 · 2024-11-18T08:47:44Z

@FxKu

Currently, I have configured the operator using topologySpreadConstraints and Affinity.PodAntiAffinity together. My expectation is that the pods are always scheduled in different nodes and availability zones. This is my manifest for them.

# OperatorConfiguration
configuration:
  kubernetes:
    enable_pod_antiaffinity: true
    pod_antiaffinity_preferred_during_scheduling: false
    pod_antiaffinity_topology_key: kubernetes.io/hostname
    topology_spread_constraints:
    - max_skew: 1
      topology_key: topology.kubernetes.io/zone
      when_unsatisfiable: DoNotSchedule

Do you want me to put that scenario in the e2e test?

laiminhtrung1997 · 2025-02-25T14:30:34Z

Excuse me @FxKu

Is there any news on this?

laiminhtrung1997 · 2025-03-08T08:09:36Z

Hi @FxKu

I have fixed all your comments. Please review the PR again. I look forward to hearing from you.

Best regards.

zalando-robot · 2025-09-07T03:35:25Z

Cannot start a pipeline due to:

No accountable user for this pipeline: no Zalando employee associated to this GitHub username

Click on pipeline status check Details link below for more information.

FxKu · 2025-10-15T13:42:24Z

pkg/cluster/k8sres.go

+	if len(topologySpreadConstraintsSpec) > 0 {
+		podSpec.TopologySpreadConstraints = generateTopologySpreadConstraints(labels, topologySpreadConstraintsSpec)
+	}
+


Suggested change

if len(topologySpreadConstraintsSpec) > 0 {

podSpec.TopologySpreadConstraints = generateTopologySpreadConstraints(labels, topologySpreadConstraintsSpec)

}

podSpec.TopologySpreadConstraints = generateTopologySpreadConstraints(labels, topologySpreadConstraintsSpec)

you only do a range loop inside the function so this len check is not necessary

I updated it.

FxKu · 2025-10-15T13:52:56Z

Can you please provide

an example in the complete manifest (somewhere here)
support in helm chart
add documentation behind this paragraph

FxKu · 2025-10-15T14:08:39Z

e2e/tests/test_e2e.py

+            }
+        }
+        k8s.api.core_v1.patch_node(master_nodes[0], patch_node_label)
+        k8s.api.core_v1.patch_node(replica_nodes[0], patch_node_label)


I would expect that the e2e test patches the Postgresql manifest and adds topologySpreadConstraints to then check if the pods spread evenly. But you're only patching the nodes here?

laiminhtrung1997 requested review from FxKu, Jan-M, hughcapet, idanovinda, jopadi, macedigital and sdudoladov as code owners February 4, 2024 15:34

laiminhtrung1997 force-pushed the add-topologySpreadConstraints branch 2 times, most recently from b948c94 to 44fcb1f Compare February 4, 2024 16:16

laiminhtrung1997 mentioned this pull request Feb 4, 2024

Add topologySpreadConstraints configuration. #2474

Closed

FxKu added this to the 2.0 milestone Feb 13, 2024

FxKu modified the milestones: 2.0, 1.13.0 May 24, 2024

FxKu reviewed Jun 26, 2024

View reviewed changes

FxKu modified the milestones: 1.13.0, 1.14.0 Jun 26, 2024

laiminhtrung1997 force-pushed the add-topologySpreadConstraints branch 8 times, most recently from 530f847 to 18023cb Compare July 7, 2024 06:56

laiminhtrung1997 force-pushed the add-topologySpreadConstraints branch 2 times, most recently from 3e99f92 to 794f6db Compare November 14, 2024 10:31

laiminhtrung1997 force-pushed the add-topologySpreadConstraints branch from 27f69f5 to b592ce8 Compare February 25, 2025 14:30

laiminhtrung1997 force-pushed the add-topologySpreadConstraints branch 2 times, most recently from adfafef to 16caf11 Compare March 8, 2025 07:28

laiminhtrung1997 force-pushed the add-topologySpreadConstraints branch from 16caf11 to dbb6aaa Compare March 21, 2025 04:40

FxKu modified the milestones: 1.15.0, 1.16.0 Jul 15, 2025

laiminhtrung1997 added 7 commits September 7, 2025 10:35

Add topologySpreadConstraints configuration to pod spec.

263242a

Modify the UT.

63f8657

Correct the assert with the list of constraints.

89e2405

Fix UT

8c9840c

Run update-codegen.sh to add deepcopy for new field to the api.

ad4dd00

Reuse configured TopologySpreadConstraints for logical backup.

f52d9bc

Remove x-kubernetes-preserve-unknown-fields and XPreserveUnknownFields.

a40ba6e

laiminhtrung1997 force-pushed the add-topologySpreadConstraints branch from dbb6aaa to a40ba6e Compare September 7, 2025 03:35

FxKu modified the milestones: 1.16.0, 1.15.0 Sep 17, 2025

Merge branch 'master' into add-topologySpreadConstraints

3a8fc1b

FxKu reviewed Oct 15, 2025

View reviewed changes

FxKu modified the milestones: 1.15.0, 1.16.0 Oct 15, 2025

Add topologySpreadConstraints configuration to pod spec. #2530

Are you sure you want to change the base?

Add topologySpreadConstraints configuration to pod spec. #2530

Conversation

laiminhtrung1997 commented Feb 4, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

monotek commented May 16, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

FxKu commented Jun 26, 2024

Uh oh!

laiminhtrung1997 commented Jun 27, 2024

Uh oh!

FxKu commented Nov 14, 2024

Uh oh!

laiminhtrung1997 commented Nov 15, 2024

Uh oh!

laiminhtrung1997 commented Nov 18, 2024

Uh oh!

laiminhtrung1997 commented Feb 25, 2025

Uh oh!

laiminhtrung1997 commented Mar 8, 2025

Uh oh!

zalando-robot commented Sep 7, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

FxKu commented Oct 15, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

laiminhtrung1997 commented Feb 4, 2024 •

edited

Loading