Right way for uploading a file from a third-party service?

[IljaN]

Hi,

I am hacking on a OpenCloud service which exposes the file system of an authenticated user via SFTP. For uploading files the InitiateFileUpload call of the Gateway-API is used with the "simple" protocol.

This works great for smaller files, however with larger files following issues arise:

• The size of a file which is still being uploaded is constantly updated in real-time, causing load.
• OpenCloud creates versions for every few hundred kb uploaded thus quickly filling up disk-space and probably impacting performance.
• The file is immediately visible to the user, despite still being uploaded.

Questions:

• Is there some kind of "staging/uploads" directory for the "simple" protocol?
• Is the "simple" protocol the right (supported) approach in this case but I am using it wrong?
• Would TUS be more suitable for "backend" file-transfers instead? From my understanding it would fix the issues above but I am not sure if it is the right protocol choice for this use case.

Thanks a lot.

[cosminmocan]

Hello @IljaN ,

While I'm currently not an active user of opencloud( that will change today however ), I have alot of experience with the other free solution.

For your second issue, isn't that the process of uploads being split into chunks, that happens on the backend when you are uploading files, specially for big files like your example.
The 3rd issue might be connected to the 2nd one, when the file is being uploaded , those chunks are being stored on you server and they are waiting for the upload to be finished, so that all the chunks can be all combined into a single file ( your original file ).

[micbar]

Hi @IljaN

I am writing from my mobile, so please excuse my short answer.

TUS is the preferred way, even for backend to backend uploads. You can see that in uppy/companion how backend uploads can benefit from TUS.

We also extended the rclone client to support TUS uploads because it is more reliable.

[IljaN]

Thanks for the hints. Looks like decomposedfs does not support TUS yet?

Hmhh, maybe the upload id could be extracted from the transfer token after the the upload is initiated via CS?

SFTP transfer is a byte stream with random access so the size of the upload is unknown in advance. Upload-Defer-Length is probably the way to go...

[micbar]

The implementation for TUS is not in the decomposedfs pkg but in the dataprovider.