implement oc chunking/checksum using tus metadata

[butonic]

I ran into checksum problems. Maybe the tus chunking extension is not what we want because it may discard whole chunks?

See clarification in tus/tus-resumable-upload-protocol#143 (comment)

all client disconnects would remove the last chunk.

Yes, that's true and one of the major drawbacks of checksum extension.

• we can send the checksum in the metadata, but for large files that would
  require reading the file twice.
• the client might calculate the checksum while sending, the server as well and can return the checksum in the response. but that would require resending the whole file in case of an error.
• actually this is the right thing to do
• together with the concatenation extension the full bandwith can be used
• but tusd does not support it, yet

AFAICT we can send the checksum in the metadata to mimic the current checksum protocol: https://doc.owncloud.org/desktop/2.4/architecture.html#upload

[butonic]

add code to cs3org/reva#669

[PVince81]

rename title s/chunking/checksum/ ?

[butonic]

AFAICT we should be good with tus/tus-resumable-upload-protocol#93

• it allows admins to specify the chunksize for uploads to bypass things like mobile iron who simply die at ~50M I was told. Treat it as rumor ...

but this resume, together with a checksum that is calculated ... during upload and sent as an HTTP trailer would cover the requirements.

I don't know what is going to die when uploading 50GB in the browser ... we need to check.

[butonic]

@michaelstingl could you comment on tus/tus-resumable-upload-protocol#93 and explain why the Tus-Min/Max-Chunk-Size header makes sense for us? It seems they are looking for good reasons to add it: tus/tusd#353 (comment)

[michaelstingl]

@butonic no need for min/max from my POV. TUS already supports fixed chunk sizes. We can have a quick call later. 🤦

[butonic]

@phil-davis @individual-it I implemented checksumming for TUS uploads in reva but I don't see acceptance tests for that in core. Checksumming when using chunks is still an open task as we need throw away any aborted PATCH request (see tus/tus-resumable-upload-protocol#143 (comment)).

[butonic]

@michaelstingl IIRC you wanted to be able to let the admin configure the chunksize to make uploads pass firewalls and intermediary services ...

[individual-it]

@butonic I don't think we have done checksums in TUS tests yet.
Please create an issue to create acceptance tests for that with a link to the PR and the part of the TUS protocol docs how it suppose to work and we can add tests for it

[butonic]

Initial Checksum support in cs3org/reva#1400