setting ALWAYS_OFFLOAD_NODE_STATUS=true in prod?

[ChenRussell]

I want to set ALWAYS_OFFLOAD_NODE_STATUS=true to reduce api server memory cost, is there any risk?

[agilgur5]

It will cause substantially more DB reads & writes (as every workflow status will be offloaded), so it's normally a trade-off if you need to relieve pressure off of the k8s api-server or etcd

[agilgur5]

Also had some prior discussion in #2333 (comment):

It is expensive to run using ALWAYS_OFFLOAD_NODE_STATUS CPU+memory+network+disk cost will all be much higher, so your AWS bills will be higher too.

and recent discussion in #7121 (comment):

1. Argo is still writing to the k8s api-server/etcd, just substantially less for the node status specifically. That means that offloading adds an additional DB. That does get quite nuanced as to when that happens and how performant each DB is respectively with large JSON payloads (see also Improve mysql write performance and stability when offloading #13290), so yes I would suggest testing the performance before/after.
2. The network latency also depends on where you host each; k8s api-server typically has very close proximity in a cluster.