Skip to content

Commit 2a67d4c

Browse files
EronWrightEronWright
authored
Set some default resource requests on the workspace pod (#707)
<!--Thanks for your contribution. See [CONTRIBUTING](CONTRIBUTING.md) for Pulumi's contribution guidelines. Help us merge your changes more quickly by adding more details such as labels, milestones, and reviewers.--> ### Proposed changes <!--Give us a brief description of what you've done and what it solves. --> Implements good defaults for the workspace resource, using a ["burstable"](https://kubernetes.io/docs/concepts/workloads/pods/pod-qos/#burstable) approach. Since a workspace pod's utilization is bursty - with low resource usage during idle times and with high resource usage during deployment ops - the pod requests a small amount of resources (64mb, 100m) to be able to idle. A deployment op is able to use much more memory - all available memory on the host. Users may customize the resources (e.g. to apply different requests and/or limits). For large/complex Pulumi apps, it might make sense to reserve more memory and/or use #694. The agent takes some pains to stay within the requested amount, using a programmatic form of the [GOMEMLIMIT](https://weaviate.io/blog/gomemlimit-a-game-changer-for-high-memory-applications) environment variable. The agent detects the requested amount via the Downward API. We don't use `GOMEMLIMIT` to avoid propagating it to sub-processes, and because the format is a Kubernetes 'quantity'. It was observed that zombies weren't being cleaned up, and this was leading to resource exhaustion. Fixed by using [tini](https://github.com/krallin/tini/) as the entrypoint process (PID 1). ### Related issues (optional) <!--Refer to related PRs or issues: #1234, or 'Fixes #1234' or 'Closes #1234'. Or link to full URLs to issues or pull requests in other GitHub repositories. --> Closes #698
1 parent 6fbcd4c commit 2a67d4c

17 files changed

+268
-97
lines changed

‎Dockerfile

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,10 @@
11
# Build a base image with modules cached.
22
FROM --platform=${BUILDPLATFORM} golang:1.23 AS base
3+
ARG TARGETARCH
4+
5+
# Install tini to reap zombie processes.
6+
ENV TINI_VERSION=v0.19.0
7+
ADD --chmod=755 https://github.com/krallin/tini/releases/download/${TINI_VERSION}/tini-static-${TARGETARCH} /tini
38

49
COPY /go.mod go.mod
510
COPY /go.sum go.sum
@@ -37,8 +42,11 @@ RUN --mount=type=cache,target=${GOCACHE} \
3742
# Use distroless as minimal base image to package the manager binary
3843
# Refer to https://github.com/GoogleContainerTools/distroless for more details
3944
FROM gcr.io/distroless/static-debian12:debug-nonroot
45+
46+
COPY --from=base /tini /tini
4047
COPY --from=op-builder /manager /manager
4148
COPY --from=agent-builder /agent /agent
4249
USER 65532:65532
4350

44-
ENTRYPOINT ["/manager"]
51+
ENTRYPOINT ["/tini", "--"]
52+
CMD ["/manager"]

‎agent/cmd/serve.go

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@ import (
2222
"os"
2323
"os/signal"
2424
"path/filepath"
25+
"runtime/debug"
2526
"syscall"
2627

2728
"github.com/pulumi/pulumi-kubernetes-operator/v2/agent/pkg/server"
@@ -30,6 +31,7 @@ import (
3031
"github.com/spf13/cobra"
3132
"go.uber.org/zap"
3233
"go.uber.org/zap/zapio"
34+
"k8s.io/apimachinery/pkg/api/resource"
3335
)
3436

3537
var (
@@ -52,6 +54,15 @@ var serveCmd = &cobra.Command{
5254
log.Infow("Pulumi Kubernetes Agent", "version", version.Version)
5355
log.Debugw("executing serve command", "WorkDir", _workDir)
5456

57+
// limit the agent's memory usage to the configured quantity (e.g. 64Mi)
58+
if limit, ok := os.LookupEnv("AGENT_MEMLIMIT"); ok {
59+
val := resource.MustParse(limit)
60+
if !val.IsZero() {
61+
log.Debugf("setting memory limit to %s", limit)
62+
debug.SetMemoryLimit(val.Value())
63+
}
64+
}
65+
5566
// open the workspace using auto api
5667
workspaceOpts := []auto.LocalWorkspaceOption{}
5768
workDir, err := filepath.EvalSymlinks(_workDir) // resolve the true location of the workspace

‎deploy/crds/auto.pulumi.com_workspaces.yaml

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -8413,8 +8413,12 @@ spec:
84138413
type: object
84148414
securityProfile:
84158415
default: restricted
8416-
description: SecurityProfile applies a security profile to the workspace,
8417-
'restricted' by default.
8416+
description: |-
8417+
SecurityProfile applies a security profile to the workspace.
8418+
The restricted profile (default) runs the pod as a non-root user and with a security context that conforms with
8419+
the Restricted policy of the Pod Security Standards.
8420+
The baseline profile runs the pod as the root user and with a security context that conforms with
8421+
the Baseline policy of the Pod Security Standards.
84188422
type: string
84198423
serviceAccountName:
84208424
default: default

‎deploy/crds/pulumi.com_stacks.yaml

Lines changed: 12 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -9334,8 +9334,12 @@ spec:
93349334
type: object
93359335
securityProfile:
93369336
default: restricted
9337-
description: SecurityProfile applies a security profile to
9338-
the workspace, 'restricted' by default.
9337+
description: |-
9338+
SecurityProfile applies a security profile to the workspace.
9339+
The restricted profile (default) runs the pod as a non-root user and with a security context that conforms with
9340+
the Restricted policy of the Pod Security Standards.
9341+
The baseline profile runs the pod as the root user and with a security context that conforms with
9342+
the Baseline policy of the Pod Security Standards.
93399343
type: string
93409344
serviceAccountName:
93419345
default: default
@@ -18871,8 +18875,12 @@ spec:
1887118875
type: object
1887218876
securityProfile:
1887318877
default: restricted
18874-
description: SecurityProfile applies a security profile to
18875-
the workspace, 'restricted' by default.
18878+
description: |-
18879+
SecurityProfile applies a security profile to the workspace.
18880+
The restricted profile (default) runs the pod as a non-root user and with a security context that conforms with
18881+
the Restricted policy of the Pod Security Standards.
18882+
The baseline profile runs the pod as the root user and with a security context that conforms with
18883+
the Baseline policy of the Pod Security Standards.
1887618884
type: string
1887718885
serviceAccountName:
1887818886
default: default

‎deploy/helm/pulumi-operator/crds/auto.pulumi.com_workspaces.yaml

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -8413,8 +8413,12 @@ spec:
84138413
type: object
84148414
securityProfile:
84158415
default: restricted
8416-
description: SecurityProfile applies a security profile to the workspace,
8417-
'restricted' by default.
8416+
description: |-
8417+
SecurityProfile applies a security profile to the workspace.
8418+
The restricted profile (default) runs the pod as a non-root user and with a security context that conforms with
8419+
the Restricted policy of the Pod Security Standards.
8420+
The baseline profile runs the pod as the root user and with a security context that conforms with
8421+
the Baseline policy of the Pod Security Standards.
84188422
type: string
84198423
serviceAccountName:
84208424
default: default

‎deploy/helm/pulumi-operator/crds/pulumi.com_stacks.yaml

Lines changed: 12 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -9334,8 +9334,12 @@ spec:
93349334
type: object
93359335
securityProfile:
93369336
default: restricted
9337-
description: SecurityProfile applies a security profile to
9338-
the workspace, 'restricted' by default.
9337+
description: |-
9338+
SecurityProfile applies a security profile to the workspace.
9339+
The restricted profile (default) runs the pod as a non-root user and with a security context that conforms with
9340+
the Restricted policy of the Pod Security Standards.
9341+
The baseline profile runs the pod as the root user and with a security context that conforms with
9342+
the Baseline policy of the Pod Security Standards.
93399343
type: string
93409344
serviceAccountName:
93419345
default: default
@@ -18871,8 +18875,12 @@ spec:
1887118875
type: object
1887218876
securityProfile:
1887318877
default: restricted
18874-
description: SecurityProfile applies a security profile to
18875-
the workspace, 'restricted' by default.
18878+
description: |-
18879+
SecurityProfile applies a security profile to the workspace.
18880+
The restricted profile (default) runs the pod as a non-root user and with a security context that conforms with
18881+
the Restricted policy of the Pod Security Standards.
18882+
The baseline profile runs the pod as the root user and with a security context that conforms with
18883+
the Baseline policy of the Pod Security Standards.
1887618884
type: string
1887718885
serviceAccountName:
1887818886
default: default

‎deploy/helm/pulumi-operator/templates/deployment.yaml

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -33,13 +33,12 @@ spec:
3333
{{- toYaml .Values.extraSidecars | nindent 8 }}
3434
{{- end}}
3535
- name: manager
36-
command:
37-
- /manager
3836
args:
37+
- /manager
3938
- --leader-elect
4039
- --health-probe-bind-address=:8081
4140
- --metrics-bind-address=:8383
42-
- --program-fs-adv-addr=pulumi-kubernetes-operator.$(POD_NAMESPACE).svc.cluster.local
41+
- --program-fs-adv-addr=pulumi-kubernetes-operator.$(POD_NAMESPACE).svc.cluster.local:80
4342
- --zap-log-level={{ .Values.controller.logLevel }}
4443
- --zap-time-encoding=iso8601
4544
env:

‎deploy/yaml/install.yaml

Lines changed: 20 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -9569,8 +9569,12 @@ spec:
95699569
type: object
95709570
securityProfile:
95719571
default: restricted
9572-
description: SecurityProfile applies a security profile to
9573-
the workspace, 'restricted' by default.
9572+
description: |-
9573+
SecurityProfile applies a security profile to the workspace.
9574+
The restricted profile (default) runs the pod as a non-root user and with a security context that conforms with
9575+
the Restricted policy of the Pod Security Standards.
9576+
The baseline profile runs the pod as the root user and with a security context that conforms with
9577+
the Baseline policy of the Pod Security Standards.
95749578
type: string
95759579
serviceAccountName:
95769580
default: default
@@ -19106,8 +19110,12 @@ spec:
1910619110
type: object
1910719111
securityProfile:
1910819112
default: restricted
19109-
description: SecurityProfile applies a security profile to
19110-
the workspace, 'restricted' by default.
19113+
description: |-
19114+
SecurityProfile applies a security profile to the workspace.
19115+
The restricted profile (default) runs the pod as a non-root user and with a security context that conforms with
19116+
the Restricted policy of the Pod Security Standards.
19117+
The baseline profile runs the pod as the root user and with a security context that conforms with
19118+
the Baseline policy of the Pod Security Standards.
1911119119
type: string
1911219120
serviceAccountName:
1911319121
default: default
@@ -27868,8 +27876,12 @@ spec:
2786827876
type: object
2786927877
securityProfile:
2787027878
default: restricted
27871-
description: SecurityProfile applies a security profile to the workspace,
27872-
'restricted' by default.
27879+
description: |-
27880+
SecurityProfile applies a security profile to the workspace.
27881+
The restricted profile (default) runs the pod as a non-root user and with a security context that conforms with
27882+
the Restricted policy of the Pod Security Standards.
27883+
The baseline profile runs the pod as the root user and with a security context that conforms with
27884+
the Baseline policy of the Pod Security Standards.
2787327885
type: string
2787427886
serviceAccountName:
2787527887
default: default
@@ -28388,14 +28400,13 @@ spec:
2838828400
spec:
2838928401
containers:
2839028402
- args:
28403+
- /manager
2839128404
- --leader-elect
2839228405
- --health-probe-bind-address=:8081
2839328406
- --metrics-bind-address=:8383
28394-
- --program-fs-adv-addr=pulumi-kubernetes-operator.$(POD_NAMESPACE).svc.cluster.local
28407+
- --program-fs-adv-addr=pulumi-kubernetes-operator.$(POD_NAMESPACE).svc.cluster.local:80
2839528408
- --zap-log-level=error
2839628409
- --zap-time-encoding=iso8601
28397-
command:
28398-
- /manager
2839928410
env:
2840028411
- name: POD_NAMESPACE
2840128412
valueFrom:

‎operator/Makefile

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -145,7 +145,7 @@ build: manifests generate fmt vet ## Build manager binary.
145145

146146
.PHONY: run
147147
run: manifests generate fmt vet ## Run a controller from your host.
148-
go run ./cmd/main.go --program-fs-adv-addr=localhost:9090
148+
WORKSPACE_LOCALHOST=localhost:50051 SOURCE_CONTROLLER_LOCALHOST=localhost:9090 go run ./cmd/main.go
149149

150150
# If you wish to build the manager image targeting other platforms you can use the --platform flag.
151151
# (i.e. docker build --platform linux/arm64). However, you must enable docker buildKit for it.
@@ -183,7 +183,7 @@ endif
183183

184184
.PHONY: install
185185
install: manifests kustomize ## Install CRDs into the K8s cluster specified in ~/.kube/config.
186-
$(KUSTOMIZE) build config/crd | $(KUBECTL) apply --server-side=true -f -
186+
$(KUSTOMIZE) build config/crd | $(KUBECTL) apply --server-side=true --force-conflicts -f -
187187

188188
.PHONY: uninstall
189189
uninstall: manifests kustomize ## Uninstall CRDs from the K8s cluster specified in ~/.kube/config. Call with ignore-not-found=true to ignore resource not found errors during deletion.

‎operator/api/auto/v1alpha1/workspace_types.go

Lines changed: 5 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -34,11 +34,6 @@ const (
3434
SecurityProfileBaseline SecurityProfile = "baseline"
3535
// SecurityProfileRestricted applies the restricted security profile.
3636
SecurityProfileRestricted SecurityProfile = "restricted"
37-
38-
// SecurityProfileBaselineDefaultImage is the default image used when the security profile is 'baseline'.
39-
SecurityProfileBaselineDefaultImage = "pulumi/pulumi:latest"
40-
// SecurityProfileRestrictedDefaultImage is the default image used when the security profile is 'restricted'.
41-
SecurityProfileRestrictedDefaultImage = "pulumi/pulumi:latest-nonroot"
4237
)
4338

4439
// WorkspaceSpec defines the desired state of Workspace
@@ -47,7 +42,11 @@ type WorkspaceSpec struct {
4742
// +kubebuilder:default="default"
4843
ServiceAccountName string `json:"serviceAccountName,omitempty"`
4944

50-
// SecurityProfile applies a security profile to the workspace, 'restricted' by default.
45+
// SecurityProfile applies a security profile to the workspace.
46+
// The restricted profile (default) runs the pod as a non-root user and with a security context that conforms with
47+
// the Restricted policy of the Pod Security Standards.
48+
// The baseline profile runs the pod as the root user and with a security context that conforms with
49+
// the Baseline policy of the Pod Security Standards.
5150
// +kubebuilder:default="restricted"
5251
// +optional
5352
SecurityProfile SecurityProfile `json:"securityProfile,omitempty"`

‎operator/cmd/main.go

Lines changed: 4 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,6 @@ import (
2727
// Import all Kubernetes client auth plugins (e.g. Azure, GCP, OIDC, etc.)
2828
// to ensure that exec-entrypoint and run can make use of them.
2929
_ "k8s.io/client-go/plugin/pkg/client/auth"
30-
"sigs.k8s.io/controller-runtime/pkg/client"
3130

3231
sourcev1 "github.com/fluxcd/source-controller/api/v1"
3332
sourcev1b2 "github.com/fluxcd/source-controller/api/v1beta2"
@@ -150,7 +149,10 @@ func main() {
150149

151150
// Create a new ProgramHandler to handle Program objects. Both the ProgramReconciler and the file server need to
152151
// access the ProgramHandler, so it is created here and passed to both.
153-
pHandler := newProgramHandler(mgr.GetClient(), programFSAdvAddr)
152+
if programFSAdvAddr == "" {
153+
programFSAdvAddr = determineAdvAddr(programFSAddr)
154+
}
155+
pHandler := pulumicontroller.NewProgramHandler(mgr.GetClient(), programFSAdvAddr)
154156

155157
if err = (&autocontroller.WorkspaceReconciler{
156158
Client: mgr.GetClient(),
@@ -246,14 +248,6 @@ func (fs pFileserver) Start(ctx context.Context) error {
246248
}
247249
}
248250

249-
func newProgramHandler(k8sClient client.Client, advAddr string) *pulumicontroller.ProgramHandler {
250-
if advAddr == "" {
251-
advAddr = determineAdvAddr(advAddr)
252-
}
253-
254-
return pulumicontroller.NewProgramHandler(k8sClient, advAddr)
255-
}
256-
257251
func determineAdvAddr(addr string) string {
258252
host, port, err := net.SplitHostPort(addr)
259253
if err != nil {

‎operator/config/crd/bases/auto.pulumi.com_workspaces.yaml

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -8413,8 +8413,12 @@ spec:
84138413
type: object
84148414
securityProfile:
84158415
default: restricted
8416-
description: SecurityProfile applies a security profile to the workspace,
8417-
'restricted' by default.
8416+
description: |-
8417+
SecurityProfile applies a security profile to the workspace.
8418+
The restricted profile (default) runs the pod as a non-root user and with a security context that conforms with
8419+
the Restricted policy of the Pod Security Standards.
8420+
The baseline profile runs the pod as the root user and with a security context that conforms with
8421+
the Baseline policy of the Pod Security Standards.
84188422
type: string
84198423
serviceAccountName:
84208424
default: default

‎operator/config/crd/bases/pulumi.com_stacks.yaml

Lines changed: 12 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -9334,8 +9334,12 @@ spec:
93349334
type: object
93359335
securityProfile:
93369336
default: restricted
9337-
description: SecurityProfile applies a security profile to
9338-
the workspace, 'restricted' by default.
9337+
description: |-
9338+
SecurityProfile applies a security profile to the workspace.
9339+
The restricted profile (default) runs the pod as a non-root user and with a security context that conforms with
9340+
the Restricted policy of the Pod Security Standards.
9341+
The baseline profile runs the pod as the root user and with a security context that conforms with
9342+
the Baseline policy of the Pod Security Standards.
93399343
type: string
93409344
serviceAccountName:
93419345
default: default
@@ -18871,8 +18875,12 @@ spec:
1887118875
type: object
1887218876
securityProfile:
1887318877
default: restricted
18874-
description: SecurityProfile applies a security profile to
18875-
the workspace, 'restricted' by default.
18878+
description: |-
18879+
SecurityProfile applies a security profile to the workspace.
18880+
The restricted profile (default) runs the pod as a non-root user and with a security context that conforms with
18881+
the Restricted policy of the Pod Security Standards.
18882+
The baseline profile runs the pod as the root user and with a security context that conforms with
18883+
the Baseline policy of the Pod Security Standards.
1887618884
type: string
1887718885
serviceAccountName:
1887818886
default: default

‎operator/config/manager/manager.yaml

Lines changed: 3 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -33,18 +33,18 @@ spec:
3333
runAsUser: 65532
3434
runAsGroup: 65532
3535
containers:
36-
- command:
37-
- /manager
36+
- name: manager
3837
env:
3938
- name: POD_NAMESPACE
4039
valueFrom:
4140
fieldRef:
4241
fieldPath: metadata.namespace
4342
args:
43+
- /manager
4444
- --leader-elect
4545
- --health-probe-bind-address=:8081
4646
- --metrics-bind-address=:8383
47-
- --program-fs-adv-addr=pulumi-kubernetes-operator.$(POD_NAMESPACE).svc.cluster.local
47+
- --program-fs-adv-addr=pulumi-kubernetes-operator.$(POD_NAMESPACE).svc.cluster.local:80
4848
- --zap-log-level=error
4949
- --zap-time-encoding=iso8601
5050
ports:
@@ -56,7 +56,6 @@ spec:
5656
protocol: TCP
5757
image: controller:latest
5858
imagePullPolicy: IfNotPresent
59-
name: manager
6059
securityContext:
6160
allowPrivilegeEscalation: false
6261
capabilities:

0 commit comments

Comments
 (0)