Improving single-key VO performance

[AhmedSoliman]

VOs single key performance can be a deal breaker for restate users. While it's true that there will always be a limit to the throughput of a single key (speed of light, network latency, etc.), we can do better than what we have today.

The primary use-case pattern that we should improve is when users want to perform a few state changes without any awaits. For instance, incrementing a counter, or sending an invocation to another service. In other words, there should be almost no reasons for our users to get better latency by batching in something like Redis.

Level 1 (stack invocations on a single stream)

• Restate opens a connection/FIFO stream to a deployment
• Send first request + eager state
• While the request is in-flight, enqueue more (up to N) requests onto the same stream but without sending state
• SDK streams responses/journal entries back as usual
• Once an invocation completes, SDK picks up the next invocation from the in-memory queue and uses the cached state from the previous invocation
• The invoker doesn't need to sync/flush/commit before sending other invocations
• If the SDK needs to await, this will automatically act as a sync/flush point.

This can be done for N invocations before switching to another stream to balance load over deployment endpoints.

Level 2 (server side services)

• WASI/Lua services that run on the partition processor. Those can only mutate state, create awakeables, send invocations, etc. No access to the outside world and limited resources are applied.
• Initially, this can be restricted to providing pre-packaged services until we feel comfortable with the direction.