Appendable s3dol

[andeaseme]

We need an incremental contiguous storage for live waveforms.

S3 has multipart uploads

Sample multipart upload calls

For this example, assume that you are generating a multipart upload for a 100 GB file. In this case, you would have the following API calls for the entire process. There would be a total of 1002 API calls.

• A CreateMultipartUpload call to start the process.

• 1000 individual UploadPart calls, each uploading a part of 100 MB, for a total size of 100 GB.

• A CompleteMultipartUpload call to finish the process.

We need an interface

Minimally implement the 3 APIs as methods.

My suggestion is to use s3bucket[key] += part. It would need to CreateMultipartUpload if it empty. Keep track of the parts uploaded. A way to complete, maybe a sentinel value. Ideally, it would be an interface that will be shared between other stores but we don't have that well established yet.

Also need a means to clear out any incomplete multipart in case we want to discard (part of a large file) and if we want to keep (part of a wf) so they aren't stuck in limbo and taking up s3 space.

[thorwhalen]

Great find. Now that I see it, I remember our team using it in several occasions. Let's not forget it this time. Or... let's indeed use this in s3dol, and then forget about it, since it's not a functionality concern, but an implementation one.

Had a discussion with @valentin-feron about it. I'll let him add his comments. Here's my part:

s[k] = v for s3

We definitely want to use it when v is big.

Do we use it conditionally (conditioned on the size of v) or systematically?
I'd say systematically (if possible) since the overhead is paid for small files.
Though if possible, I'd separate, encapsulate, and make it easy to change this default choice
(e.g. though a setitem parameter, or a mixin that has a __setitem__).

The three objects you mentioned have parameters too. Make sure the user can change these.
I'd say that we encapsulate the trio into a item_setter (or setitem?) object (callable instance of a class or partial of a function). Then, I'd give (the possibility of) control to the user through a item_setter argument or a (mk_item_setter, item_setter_params) pair.

s[k] += v for s3

Before doing this, some familiarity with the general s[k] += v use cases, problems and designs should be known. Still, with that knowledge, it's completely acceptable (and even wise) to start with the s3-specific implementation, and refactoring to use the more general tools later. (Add a TODO for that in the code.)

That said, it should be noted (as @valentin-feron did) that we don't necessarily want to use s[k] += parts_of_v to implement the s[k] = v, since unless we do some extra work (that we don't want to do at this point, unless it's quick and easy), s[k] += parts_of_v won't take advantage of the asynchronous aspects of the multipart upload.

s[k] += v in general

Yes to making a well designed, dol-wide consistent interface for this.
Be well aware of the many use-cases, tools, and attempts of the past before embarking on this.

Please spend a bit of poking around dol, py2store and creek for previous thoughts (issues, discussions, wikis) and actual code regarding this. As you read stuff, please add your comments, or links to relevant resources (internal or external) for the problem

Not sure it's the best name for it, but I already have this Paging (aka chunking, batch) tools discussion started, so lets accumulate the general ideas there, and the "s3dol in particular" here.
I added this new comment about the problem, which you should definitely read.

[valentin-feron]

Yes, s[k] = v and s[k] += v are 2 different problems:

• s[k] = v is about making the operation possible, "regardless" the size of v (and maybe other aspects), and optimize it
• s[k] += v is about extending an existing value in the store.

I think we don't want to use the latter to handle the former, but do the opposite (kind of at least). Indeed, as @thorwhalen mentioned, s[k] += v is a sequential way to append data, when multipart allow us to upload parts in parallel. Now, when we do s[k] += v, we want to be able to do it, regardless the size of v, so we also want to use multipart upload here (reuse the logic of s[k] = v).