Proposal: a new TusAndroidClient similar to TUSKit for iOS

[liyoung47]

History

Back in '23, MapQuest used TUS for a project requiring reliable uploads of user photos. We used TUSKit for iOS but found the android version missing the features we needed - most notably, set-and-forget uploading that didn't stop when the app closed - so we volunteered to contribute this work back to TUS. We met a couple times, but then priorities changed and MapQuest didn't have bandwidth for me to finish the last step, cleaning up and opening the PR. Recently tho I've spent some weekends cleaning this up - enough at least to draft this PR!

This is based on our code that's been working well in production since late '23 (but of course has new edits so its always possible I introduced a bug 🐞 )

Background

In the current version of the tus android SDK, you create a TusExecutor and a TusUploader and call uploadChunk() repeatedly in a while loop from a background task. This is kinda low-level and by itself has the problem that uploads will stop when the app is killed, with no way to resume them on the next launch.

The features of TUSKit that we used on iOS and needed in Android were:

• it persisted a copy of the file to be uploaded, so that we could retry any time
• it used iOS features to continue uploads in the background
• on next app launch we were able to find out what had succeeded and failed

What we didn't like so much:

• the upload retried only 3 times before reporting failure. We wanted near-infinite retry - we were able to get around this by getting the ids of any uploads that failed, and telling the client to retry them.

Proposed Changes

This PR introduces a TusAndroidClient class.

File Upload

Like on iOS, its set-and-forget. You can give it a file, it will make a copy locally, start the upload, retry as needed, and notify you when it succeeds. Uploads will continue in the background when the app is closed, or whenever the operating system decides is a good time, using Android's Work Manager, the "recommended solution for persistent work."

You would provide a file Uri to upload like so (also accepts an InputStream):
String id = tusAndroidClient.submitFileForUpload(uri);
There are versions of this method which accept metadata and custom headers, too.

The submitFileForUpload method still needs to be called from a background thread - which is maybe something we should change as its a little unexpected for the high-level TusAndroidClient. For our purposes we already were on a background thread (compressing images before uploading) so it was convenient to have it that way.

Retries

TusAndroidClient does not have an explicit way to retry like iOS does. Its built to support infinite retries, or allows you specify either a maximum number of retries or a date at which to stop retrying (In this way, its more flexible than TUSKit). It is possible that an upload will fail in a way that cannot be retried - we call this a permanent failure. This occurs if we become unable to access our local copy of the file to upload, or if uploadChunk() throws a ProtocolException where shouldRetry returns false.

A SubmissionPolicy can be created and passed along to submitFileForUpload - this lets you specify the stop date, max retries, backoff strategy (linear or exponential) and also upload criteria: whether the upload should happen on any network, or only on unmetered networks.

Listening to changes

The UploadStateInfo class is used to provide a complete picture of the current state of uploads. It contains 3 maps. The keys are upload ids and the values holds data about the upload. One map is for successful uploads, one map is for pending uploads (either in-progress, or scheduled), and one is for permanently failed uploads. The available data on an upload depends on its state. For all uploads we provide the upload metadata. For permanently failed uploads we provide failure reason and for pending uploads, we include progress in the range 0-100.

There are several ways to receive updates.
You can get a one-time update any time by calling getPendingUploadInfo and supplying a callback (a callback is required because we may not have the information right away, we are dependent on receiving a status update from WorkManager.)

You can also addPendingUploadChangeListener and receive a new UploadStateInfo whenever anything changes (including any time there is a change in the progress % of a pending upload).

You can also receive a one-time notice of each upload that succeeds with addUploadSuccessListener and each upload that permanently fails with addUploadFailureListener. You will only be notified if the success or permanent failure happens while the app is running.

The keys of the maps in UploadStateInfo are the same ids that are returned from submitFileForUpload so if you wanted you could hold on to the id and check on the status of a particular upload by seeing which map contains that key. Its not something we found the need for; in practice we used the metadata to identify a particular upload.

To avoid wasting memory, information about succeeded and permanently failed uploads is cleared. As a result, information about these uploads is only available during the session in which they succeed or fail, or on the next app launch (if they succeed/fail while the app is not running). In contrast, information about any still-pending upload is always available. So if an upload succeeds while the app is running, that upload will appear in UploadStateInfo for that session, but when the app is killed and relaunched, the new UploadStateInfo will no longer contain any information about the successful upload from last time.

Implementation Notes

Files are copied locally to the directory specified when you initialize TusAndroidClient.

Each upload has two Worker tasks, a TusUploadWorker which does the actual upload including calling uploadChunk() in a while loop, and FileCleanupWorker which runs only when TusUploadWorker succeeds, and deletes the local copy of the file.

SharedPreferences is used to store data from the workers and read it in the TusAndroidClient. I didn't allow the user to specify a custom SharedPreferences location when creating TusAndroidClient because it needs to be the same SharedPrefs each time in order for the stored metadata from workers to be read.

Outstanding Questions

In this PR I have removed TusUploader, it seemed unusual to expose that when we now have the TusAndroidClient? I noticed we were in version 0 before and wondered if a breaking change like this would be acceptable, possibly going to version 1 with these changes?

Work still to do

• Threading
  Possibly want another version of submitFileForUpload that you don't need to call from background thread?
• Kotlin example
  Had I written this today, I would have used Kotlin instead of Java. At least we ought to have a Kotlin example activity as well as the one in Java.
• Testing
  More can be done to unit test the workers as described here

Next steps?

Hopefully this is useful for you and maybe it can become the next version of tus android? I'd be excited to hear your thoughts/feedback!

[Acconut]

Thank you very much for this incredible PR! I remember us having a chat about tus-android-client some time ago and it's great to see the result of your work!

As you mentioned, tus-android-client current interface is rather cumbersome to use, especially compared to other libraries we maintain, such as tus-js-client or TUSKit. In fact, each library has its own, unique interface, causing friction for people who want to use tus in multiple different projects and us as maintainers as well. Therefore, we've been kicking around the idea of revamping our tus clients to be a bit more similar in the way they operate.

I've been especially looking at tus-java-client, which forms the basis for tus-android-client, as it appears to need the most work. Therefore, I think for all changes to tus-android-client, we should think about whether can be made to tus-java-client as well. Of course, some aspects don't make sense in tus-java-client, such as background uploading, but others could be relevant, such as progress reporting.

In the past, I've looked at tus-android-client as providing a handful of Android-specific utilities to be used next to tus-java-client. The benefit is that all improvements to tus-java-client equivalently apply to tus-android-client users as well. This PR breaks with this approach by introducing the TusAndroidClient class, which isn't an addition to tus-java-client but rather a library built on top of tus-java-client.

While I don't want say that this isn't a valid approach, I'm worry of its implication. Namely that it introduces another interface and essentially another library that we have to maintain. Therefore, I would like to consider evolving tus-android-client and tus-java-client together.

• it persisted a copy of the file to be uploaded, so that we could retry any time

This is an interested idea. I've mostly worked with tus-js-client so far, where saving a copy of the file is usually not possible due to restrictions in browser environments. However, this also means that tus-js-client users have to figure out their own solution to obtain files when they want to resume their uploads. Saving a copy of the input file does help here. However, this might cause problems with disk space when large files are duplicated. In addition, I'm curious how InputStreams are handled? Are they fully copied to disk before the upload begins? I would be curious to hear what your experience with these two points have been in the past.

• he upload retried only 3 times before reporting failure. We wanted near-infinite retry

Tus-js-client will retry whenever it made progress regarding the amount of transferred data since the last error. Effectively, it doesn't have a hard retry limit but will retry as long as the upload is progressing even in the presence of errors. Only if it doesn't see progress, it will stop retrying after a fixed retry count. I think a similar approach would make sense here as well, allowing uploads to complete if possible while also not ending up in endless loop.

Overall, I'm very grateful for this PR and I'm curious on your input on the idea of moving some of these improvements to tus-java-client!

[liyoung47]

Hi @Acconut, thanks for taking a look! Sorry to be so slow getting back to you, I was expecting to hear back on slack and didn’t see your message until now (reminder for me to change my notification settings!)

revamping tus clients to be a bit more similar

That would be awesome, the differences between iOS and Android were definitely a stumbling block for us. It would be great if all the clients were more similar in what they do for you: do they store the file? do they retry automatically and how much of that can you control? can you pause or cancel? (TusAndroidClient is currently lacking the ability to pause/cancel).

There’s also some things you’d expect in an Android or iOS sdk even tho there’s no equivalent in other sdks, like using the platform-specific mechanisms like WorkManager on Android (or a background URLSession on iOS like TUSKit does) so that the upload doesn’t stop when the app gets killed, and the os can schedule it to happen at the right times (like when a network connection becomes available).

evolving tus-andoid-client and tus-java-client together

It was actually my original plan to rework tus-java-client first and make it do all the key features we needed, with interfaces that we could use to slot in actual android-specific implementations (like using SharedPreferences instead of, say, reading and writing to a custom file on the java version to persist the state of in progress uploads across launches).

When it came to implementation however I realized we’d likely need the structure inside to be quite different, which wouldn’t lend itself well to that approach.

If we were implementing a TusAndroidClient-equivalent for tus-java-client, with a simple submit method, automatic retries, and callbacks for success and failure, I’m thinking we’d want to implement it using a pool of java threads. When a new file is submitted, we’d spin up a new thread from our pool to first copy over the file to our storage directory, and then do uploading (network connection/chunking/while loop). If the upload failed we could schedule a new thread to resume the work after the delay we wanted. This should work great for a plain java program, allowing the uploading of each file to run independently and without blocking program execution.

That wouldn’t translate well to Android however. Although you can launch threads inside Android it would be the wrong approach in an environment where network is unreliable and the application can be killed at any time. For android file uploads you really want WorkManager and to use it you must have the work take place inside a subclasses of Worker, which you communicate with only thru data which can be serialized (that’s so the operating system can run, pause or restart a worker whenever phone conditions are right, without relying on any other code from the rest of the app. The rest of the app may not even be running at the time).

To hide the use of Worker vs Thread, I think the most we could say about the common interface between Android and Java is that it lets you submit files to upload, and has a way to emit progress updates on those uploads. That’s actually what TusAndroidClient does. So for java we could implement an equivalent class that adheres to the same interface but implements it using threads.

One bit of code that I think we could and should try to share between java and android is the uploading logic that lives inside TusUploadWorker.doWork() which is also what’s currently in the tus-java-client example app. It would do a similar function whether called from inside a Thread or a Worker: it runs, it may be stopped, it may fail, and it needs to report and save its progress and status. I think we should be able to use interfaces to hide details of how it is interrupted and how it saves progress on each platform.

saving a copy of the input file

I think it does make it easier, I might even say that unless you save it, it feels like resuming is only half-implemented? The sdk remembers where we were with the upload but we have to find the file we were uploading, which the user may have moved/deleted meanwhile. Or maybe it was only ever in-memory to begin with.

For input streams, yes they are fully copied over to the storage directory on a background thread before we start the upload worker.

Disk space is a valid concern. For our particular use case the files we’re uploading are minimized to start with, and are critically important to the user so we just leave them there until upload succeeds. We’ll be notified if the system is low on memory, and we were planning to delete them ourselves in that case (Actually that was a post-MVP feature, we haven’t done it yet and we haven’t had any problems reported from users in the wild these last 2 years). Since the sdk user is giving us the storage location, you could argue it’s up to them to monitor the memory usage and manage that if they want. By way of managing, it would be nice if we gave users a way to cancel uploads (which TusAndroidClient doesn’t do yet).

Maybe in some clients it makes sense to give users the choice of having it copied and saved, or not. I think there’s not much we could do with an InputStream if we don’t copy it over. We use it now, and it’s gone when we want to retry. But if we are given a Uri we wouldn’t necessarily need to copy it, we could persist the uri path and try to read from it again later for the retry. If it’s no longer accessible then of course the retry fails.

retry as long as the upload is progressing

That sounds like a good criteria for retrying in general. It might be a problem for MapQuest’s use case however. Since the files we’re uploading are so important to our customers, we wouldn’t want to risk declaring permanent upload failure and discarding them just because we’re not making progress for a while (maybe network connectivity or a backend outage could make the upload keep erroring while not progressing).

Could we offer that as additional strategy among several that could be specified? So users could either specify a time limit, or a max retry count or “retry as long as progress is made”. Maybe these could even be combined, e.g I want to retry as long as progress is made, for a maximum time of 5 days. The more control we give users, the better it could fit whatever requirements their application has.

Also curious do your other clients have the concept that some failures are permanent and should never retry? I’m using ProtocolException’s shouldRetry() method and declaring permanent failure with no retries if that returns false. It looks like it always returns false unless the error is a 5XX or 423.

Anyway, thank you so much for your time and feedback! Glad I was able to get this up so you at least have the code to use if you want. Let me know what you think especially about my proposed tus-java-client implementation and what we can share and not share with tus-android-client. If this sounds in line with what you’d like to do, I can try implementing the java version with threading. (Not for work, just in my free time so no guarantees how fast this would get done! 😁 )