does the plugin uses s3 as an append-only database?

[fyears]

(I am the author of remotely-save)

To my surprise livesync adds support for s3 recently. So (obviously) out of curiosity I give it a try. :-)

It seems that lots of jsonl.gz files are generated in s3. So does live sync uses s3 as an append-only database, instead of raw file storage?

[berot3]

Any news?

[vrtmrz]

Wow! I am very honoured to receive your message! I would love to express great respect and appreciation to you and your codebase. It has so helped me! However, sorry for being quite late in replying!

It seems that lots of jsonl.gz files are generated in s3. So does live sync uses s3 as an append-only database, instead of raw file storage?

Yes, Self-hosted LiveSync uses an S3 bucket as an append-only storage, like to save journals.
The files sent and received do not represent files. It consisted of the metadata of the files or the differences that have occurred.

Self-hosted LiveSync splits the file into several chunks. These chunks have the id determined by its content. And, metadata keeps the list of chunk IDs instead of its content. So we only have to exchange missing chunks and journals.

Fortunately, here is a design document. I hope that this helps us.

[muhlba91]

@vrtmrz quick question for understanding this append-only process as one who's not familiar with couchdb: does this mean that the storage needed in S3 (and in pouchdb locally?) is growing over time without compaction or cleanup of unneeded/old chunks?

[vrtmrz]

I am writing an explanation, but it is taking longer than I expected, so I will reply to you first.

This is as you say. It increases all the time.

There are several reasons, but the biggest one is that conflicts are always 'found to have happened' in previous revisions from the latest at the present, when have synchronised.

This is where we differ the most from other synchronisation solutions. Because of this characteristic, we can smoothly discover the revisions that are the base of differences and merge their differences as long as there are no inconsistencies.

The feature such as displaying history are also derived from. However, the key is conflict resolution.