Scalability: move ephemeral state to redis

[knolleary]

Part of #2782

Whilst the work under #2782 has removed most in-memory state handling, there are some pieces that are not suitable for managing via the postgres database.

• API Rate-limiting - state is currently held in memory. The module we use supports connecting to redis out of the box (or a redis-api compatible alternative)
• Instance/Device inflight state - when an instance/device is transitioning between states, we store some state in memory and some in the database. Need to review what, if anything, would be more suited to holding separately.

Tasks - Phase One

• Identify what ephemeral state we still have
• Create 'cache' component in forge app to centralise via API
• Update forge code to use cache component in place of local in-memory state

Tasks - Phase two

• Identify what technology to use (redis, or an alternative given recent licensing changes)
• Add to our helm charts
• Setup authentication mechanism
• Update rate-limiting to use redis if configured
• Review instance/device lifecycle to see if any other state needs moving in

[hardillb]

Or some suitable fork/alternative (though current new license only impacts SaaS offerings of Redis iirc) e.g.

• dragonflydb
• keydb
• memcache

[hardillb]

First quick pass on existing code

Forge App

• forge/settings/index.js Platform settings are loaded at startup and only updated if the instance changes them. They are never re-read from the db, the object could be shared or reads could pull from the database.
• forge/db/controllers/Projects.js includes inflightProjectState (object), latestProjectState (object), inflightDeploys (Set)
• forge/comms/devices.js keeps a list of inflight commands, and device logs websocket clients, would need to ensure that session affinity is used here and heartbeats are in shared cache

Drivers

• driver-localfs is not intended to be used with multiple instances (keeps used ports and file handles for logs in memory)
• driver-docker not intended to be used with multiple instances, does keep an internal in memory list of project states (this._projects) used to shortcut methods for suspened instances
• driver-k8s has the same internal project state cache (this._projects), mainly used to shortcut methods to return instantly if not found. would need to move to shared cache

I need to dig a little harder to make sure I've not missed any.

@cstns is all the ephemeral state you added recently just in the Projects controller?