Delete default keys from podTemplate to change volume type of "share" volume

[luckyf]

What happened?

Hi all,

due to some restrictions in the ephemeral storage of my K8s clusters, I need to change the volume type of the "share" volume in the Pulumi Stack podTemplate from emptyDir to volumeClaimTemplate (or any other volume type).
To make the examples shorter and easier to read, I used persistentVolumeClaim instead of volumeClaimTemplate.

My first try was to overwrite the volume definition like this

# spec.workspaceTemplate.spec.podTemplate.spec
volumes:
  - name: share
    persistentVolumeClaim:
      name: xxx
  - name: additionalvolume
    # ...

but this results in the following object after getting merged with the default template. As K8s cannot derive which volume type to choose, it will not schedule the pod.

# spec.workspaceTemplate.spec.podTemplate.spec
volumes:
  - name: share
    emptyDir: {} # <= This will be merged from the default workspaceTemplate
    persistentVolumeClaim:
      name: xxx
  - name: additionalvolume
    # ...

After some research on #774 and the Strategic Merge Patch pattern I tried it like this but the $patch field does not match the CRD schema and therefore cannot be applied.

# spec.workspaceTemplate.spec.podTemplate.spec
volumes:
  - name: share
    $patch: delete
  - name: share
    persistentVolumeClaim:
      name: xxx
  - name: additionalvolume
    # ...

Error message:
Error from server (BadRequest): error when creating "stack.k8s.yml": Stack in version "v1" cannot be handled as a Stack: strict decoding error: unknown field "spec.workspaceTemplate.spec.podTemplate.spec.volumes[0].$patch", unknown field "spec.workspaceTemplate.spec.podTemplate.spec.volumes[1].volumeClaimTemplate"

Is there any way to overwrite the volume type of the "share" volume inside of the podTemplate-Spec?
Thanks a lot in advance for the feedback.

Example

Expected Result:

# spec.workspaceTemplate.spec.podTemplate.spec
volumes:
  - name: share
    persistentVolumeClaim:
      name: xxx
  - name: additionalvolume
    # ...

Output of pulumi about

not relevant

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[EronWright]

Hi @luckyf and thanks for the explanation. This seems like a pretty unusual situation because the system is very much designed to use a emptyDir volume. I've never heard of problem with this type of volume, could you elaborate?

[luckyf]

Hi @EronWright,
thanks for the response.
The K8s environment I am using is pretty restrictive in limiting the nodes ephemeral storage size which the emptyDir volume type is using. This results in an envicted workspace pod during download of the pulumi packages.

To avoid this issue, I'd prefer using my own PVC, based e.g. on the StorageClass gp2 in AWS.

[EronWright]

@luckyf the problem I see with your suggestion is that the system relies on having a clean volume when the workspace pod is created. The system deliberately terminates the workspace pod to clean up the volume. Your suggestion of using a PVC would undermine that aspect of the design.

Perhaps a good solution is to set requests and limits on the workspace pod for local ephemeral storage:
https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#setting-requests-and-limits-for-local-ephemeral-storage

I'm closing this ticket now as "won't fix" in the sense that a PVC will not be supported here because ephemeral storage is what the system is designed for. Happy to keep looking for ways to ensure that the emptyDir is effective, and we can re-open this ticket if we identify a change along those lines.