Helm chart

[Reversing0148]

Moving here from the pull request #1206 as the helm chart needs improvement and I acknowledge that the pr was premature from my end.

I currently maintain my own version here.

My goals with the helm chart are three fold

1. Make it easy for newcomers to try both standalone and highly available rauthy in kubernetes,
2. Make the helm chart minimal in terms of complexity to ensure easy maintainability.
3. Make no compromises on security and adhereing to best practices documented in the Rauthy Book.

To achieve this I came up with the idea for generating a simple rauthy configuration based on the Minimal Production Configuration. I found that it is achievable without additional user input other than what is already required by the helm templates.

Reflecting on the comments in #1206 :

The server.trusted_proxies should maybe be changed to a mandatory value you must set, instead of specifying all private networks as allowed. Rauthy always tries to have as safe as possible defaults and this allows quite a big range. I would prefer the user needing to set this explicitly. An automatic extraction from the cluster during install is probably not really feasible because of the many different possibilities where / how such a range could be specified.

While I acknowledge this is a large range, even though the secret generation is based on the minimal production configuration it is never recommended for production use. This is made clear in the values.yaml and the NOTES.txt as well. Making this configurable would contradict my first goal.

I would probably choose shorter ENC KEY IDs, something more in the range of 6 or 8 characters is easier to read and identify. Or even use yyyy-mm-dd. I would also rename the value in the chart from randAlphaNum16 to something like encKeyId.

I will go ahead and implement this 1:1, using shorter, 6 character ids using the encKeyId variable name.

webauthn.webauthn does not really need to be hardcoded to Rauthy, as it has such a default already.

I found during testing that the container failed to start as this configuration was missing, I will double check and remove it if necessarry.

I only created very simple Helm charts for testing in the past. Is there a way to force an odd replicaCount directly inside the chart? It is not recommended to use odd replicas, but more like a hard requirement, as even numbers don't make any sense at all and can give a false sense of safety if someone does not know the details of Raft. This should probably be updated in the doc for the value as well.

Yes implementing validations is possible, and is probably really easy with the mod helm template function. I am happy to implement this, however validations should not get out of hand as that would contradict the second goal.

A pod affinity should absolutely be added, as this is crucial for proper HA. I currently use:

Certainly I use this myself. This can be templated into the pod template definition instead of manual configuration in the values.yaml

Maybe even add an option for the Values.yaml to specify the instance size and then set proper values here would be a nice addition. These sizes could kind of be copied from https://sebadob.github.io/rauthy/config/tuning.html#memory-allocator. For small instances, I would only allow a single hash thread and MALLOC_CONF=abort_conf:true,narenas:1,tcache_max:1024,dirty_decay_ms:1000,muzzy_decay_ms:1000. This would lower the required minimum memory to ~164MiB without sacrificing security by lowering hashing.argon2_m_cost.

I am lacking jemalloc knowledge however I am happy to implement default request and limit resources (small, medium, big and open) and their corresponding MALLOC_CONF env var into the pod template. I would also like to add a custom option, where every value is user configurable.

I would change the PodDisruptionBudget and enable it by default. Also remove the minAvailable and instead always use maxUnavailable: 1 which is a way safer setting. minAvailable of 2 for instance would kill the cluster during restarts if you use 5 replicas.

Enabling pod distruption budget by default is not the best idea in kubernetes, when the replica count is 1. This would remove the possibility to run rauthy in standalone mode. However I agree that the template should suggest minAvailable replicas-1. In a coming change I will make the pdb apply when there is more than 1 replica, with the suggested minAvailable configuration.

We could maybe add a template for providing a manual config, that could be copied from the repo.

A secret template can be added to the readme with instructions.

In addition to the above I plan a few more improvements mostly around the readme and notes. I will post an update here once I have progress to share.

A question from my end, should the helm chart version follow rauthy`s version or should it be decoupled?

[sebadob]

While I acknowledge this is a large range, even though the secret generation is based on the minimal production configuration it is never recommended for production use. This is made clear in the values.yaml and the NOTES.txt as well. Making this configurable would contradict my first goal.

Got that, yes, but it also conflicts with your 3rd goal, so this is not that easy. I am thinking about a good way to solve this. It should at least be configurable and maybe have a default value with the very broad range. Maybe having an "advanced" section or something like this in the values would be a solution.

I found during testing that the container failed to start as this configuration was missing, I will double check and remove it if necessarry.

That is true for rp_id and rp_origin, as these are crucial. The rp_name is fully optional ans even has "Rauthy IAM" as default.

(...) however validations should not get out of hand as that would contradict the second goal.

I absolutely agree. It's impossible to make something completely fool proof for people deploying something and don't even read the docs, while expecting it to run perfectly smooth. The replicas however is a pretty important thing I would say, but I would also be fine with leaving this as a note in the comment above the value.

Certainly I use this myself. This can be templated into the pod template definition instead of manual configuration in the values.yaml

Yes, hardcode directly into the config and don't even make it configurable, as it makes no real sense to ever modify this.

I am lacking jemalloc knowledge however I am happy to implement default request and limit resources (small, medium, big and open) and their corresponding MALLOC_CONF env var into the pod template. I would also like to add a custom option, where every value is user configurable.

Yeah of course. You don't need any knowledge about it, the env var can be copy & pasted as it is. Just adding this makes quite a big difference, because you can run a small instance with either 50mb (with Hiqlite), or even go up to ~150mb with a tuning in the wrong direction. The default will end up around 100mb, but I guess most people are fine with the "small" tuning. A Pod using Postgres with it will usually even be < 30mb.

Enabling pod distruption budget by default is not the best idea in kubernetes, when the replica count is 1. This would remove the possibility to run rauthy in standalone mode. However I agree that the template should suggest minAvailable replicas-1. In a coming change I will make the pdb apply when there is more than 1 replica, with the suggested minAvailable configuration.

Oh yes of course. I always deploy HA, forgot about that. But that's a good solution.

A question from my end, should the helm chart version follow rauthy`s version or should it be decoupled?

Good question. There are good reasons for doing it and not doing it. I think following Rauthys versions makes the most sense though, at least as soon as the basic chart config is stable, and it's not too far away from that. Following Rauthys versions makes breaking changes in the chart only possible, when Rauthy bumps at least the feature version, but on the other hand, when the chart bumps the version, you will know immediately that it's probably a good idea to look at Rauthys release notes to not miss something.

---

I like the idea of having a basic config inside the values, and for all more advanced deployments, providing a template with the full Rauthy config. I would probably add a link to the Reference Config so people can copy & paste it. This keeps the chart / template maintenance low.

[Reversing0148]

Just pushed an update with implementations of most of the suggestions.
Reversing0148/rauthy-helm
Key changes:

• trustedProxies is now mandatory to set for proxy mode when generating rauthy config secret,
• by default podAntiAffinity using topologyKey: kubernetes.io/hostname is preferred with weight 100,
• reworked resources block,
• malloc configuration can now be applied from presets, with support from custom value,
• implemented validations for odd replicas and other generic requirements
• removed rp_name from generated config
• podDistruptionBudget is created when replicaCount > 1

Others:

• headless service is only created, when replicaCount > 1,
• chart now comes with values schema json

As for including the config in the values, I don't think its a good practice and would prefer to avoid it because it is GitOps anti pattern. If the deployment, or infrastructure of rauthy is automated, (meaning the values.yaml is most likely in a git repostitory) storing secrets and more specifically encryption keys becomes an issue. There is already 3 ways for the user to configure rauthy.

1. Generate the secret via the values.yaml
2. Bring their own secret, or migrate from a previously generated secret,
3. When either of the above is satisfied, new configurations can be tested "quickly" by using environment variables to overwrite values.
   I found these enough for both testing, and running a HA "production" cluster.

I am curious to read your thoughts!

Edit: updated changes list

[sebadob]

I really like these changes.

Secrets should never be added to version control, I fully agree. I do this for some things, but I then use sealed secrets. However, it's unclear if Broadcom will do a rug pull for these too, but for now they work fine.

Regarding having config in the values, there are pros and cons, as always. It could be done in a way that secrets must be added manually to prevent them being checked into version control, which would make this work. But on the other hand, there are many sensitive things inside Rauthys config and I lately started providing the whole config as a secret and having nothing at all in a CM. This makes managing it a lot easier and is safe to do, compared to how it was in the beginning. I think providing a link for copy & paste with the first lines to set up the correct secret for providing a manual config is the best way here. Doing it this way, you could also check it into git when using a sealed secret for instance and it provides the most flexibility.

So, the only thing I would add is probably to the documentation for externalSecret and add the secret template like so:

apiVersion: v1
kind: Secret
metadata:
  name: rauthy-config
  annotations:
    helm.sh/resource-policy: keep
  labels:
    # add default labels 
type: Opaque
stringData:
  config.toml: |-
    # paste and adjust config from 
    # https://sebadob.github.io/rauthy/config/config_minimal.html 
    # or
    # https://sebadob.github.io/rauthy/config/config.html