Reset invocation

[slinkydeveloper]

Fix #2890, depends on #2904

[slinkydeveloper]

E2E test restatedev/e2e#377

[tillrohrmann]

Thanks for creating this PR @slinkydeveloper. I think this is a very powerful feature that our users will like a lot :-) Before we can merge it we need to make sure that there isn't a problem with modifying the JournalMetadata while a concurrent retry is happening in the invoker. I suspect that there is a possibility that the invoker might read the JournalMetadata from before a trim operation and the journal from after a trim operation because it is not using a snapshot of RocksDB. Before this was never a problem because modifying the JournalMetadata stopped the retry mechanism (via the JournalTracker) but with the time travel command this might have changed.

[slinkydeveloper]

Followup PRs will be:

• Replace admin API bifrost proposals with RPC client, to enable propagating errors back
• Store which restate version created and modified the invocation status #3197
• Add knob to Kill child invocations when resetting an invocation
• Revert state changes when resetting an invocation
• Align Kill with Reset invocation (this Trimming journal #2890 (comment))

[tillrohrmann]

Thanks a lot for updating this PR @slinkydeveloper. The changes look good to me. I mainly had minor comments. The ones that need some clarification are the following:

1. What's the backwards compatibility story if this feature gets activated in the next released version? Alternatively, we could make the feature opt-in by waiving the right to go back. For the latter, how would this be enforced?
2. We seem to always remove signals that are beyond the trim point. Aren't signals used by awakeables to complete them? If yes, can't it happen that we truncate the completion for an awakeable that won't get truncated? If yes, then we might end up being stuck because we might not re-run the logic to tell some other service to complete the awakeable again.
3. Maybe it's worth unifying a little bit the terminology we are using. The feature to reset the journal is called "reset". Reset has a "truncate_from" index which gets translated into a TrimInvocation command which causes the journal to be rewritten. Maybe we can unify reset/truncate/trim or truncate and trim?

[slinkydeveloper]

Will rebase this PR on #3227 when ready.

[slinkydeveloper]

After some internal conversations, we agreed on for now merging this feature without reverting state, and give it a try like this. This feature also needs to retain the old journals, something i'll do for #3347 primarily and then re-adapt this PR here. I'm gonna rebase few things, figure out retaining old journals, and then i'll ask another review for this PR.

[slinkydeveloper]

I rebased this PR on #3360, it also contains the changes from #3362 (1:1, so once that one is merged, it should disappear from this diff). This got a bit messy with the different branches :D For this feature specifically, only the last commit matters.