Split huge update action groups/transactions into smaller ones

[fm3]

Currently, annotation update transactions can become very big. They are split into different requests, and that works fine for the most part. But in the backend, they are recombined into a single group.

Now we got some problems with this group becoming too big. The backend tried to serialize it to json (after dropping volume base64 data, and adding authorId and timestamp to every action) and the result was >2GiB, which the JVM unfortunately still won’t do.

More context: https://scm.slack.com/archives/CMBMU5684/p1753089004459379?thread_ts=1752914209.971799&cid=CMBMU5684

One option would be to split this into multiple transactions already in the frontend, or in the backend when concatenating the transaction’s update groups. However, it is not trivial to decide where to cut, as the json string length isn’t known in advance.

Opinions on this @philippotto @MichaelBuessemeyer @normanrz ?

[normanrz]

Why is the backend recombining the split groups into a single group?
Could we also use a denser serialization than json (e.g. protobuf)?

[fm3]

Why is the backend recombining the split groups into a single group?

A transaction is a semantic unit (i.e. one user interaction or a couple that belong together). We already split a transaction into multiple requests, but the server puzzles them back together in order to store this update group as a single version in FossilDB. That way, the one user action creates one new annotation version.

Could we also use a denser serialization than json (e.g. protobuf)?

Yes, but that would come with its own set of advantages and disadvantages
+ smaller, would shift this problem towards even larger groups
+ presumably faster serialization and deserialization
− would mean a lot of additional code because of conversion to the generated classes.
− backwards compatibility: we’d have to either migrate existing update actions in FossilDB or provide both deserialization strategies for all action types, and either attempt both or encode somehow which encoding is used for a particular value in fossilDB.

It would be interesting to know what the users actually did to create such big update groups. If a single or few user interactions can create such a bulk of update actions, maybe we could instead introduce a new type of update action that expresses the same change.

[normanrz]

It would be interesting to know what the users actually did to create such big update groups. If a single or few user interactions can create such a bulk of update actions, maybe we could instead introduce a new type of update action that expresses the same change.

That would be best. Do you know what the update action was doing?

[fm3]

Unfortunately not, since it was already the saving that was rejected. I also don’t think we have an annotation id.