From 36e8ad8ab2d9f1e1ea52a84b7671dc397b8744f6 Mon Sep 17 00:00:00 2001 From: Felix Schwarz Date: Tue, 23 Feb 2021 11:55:41 +0100 Subject: [PATCH 1/6] - First draft - The base64 encoded parts of the example requests and responses is invalid fake data at that point --- protocol.md | 113 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 113 insertions(+) diff --git a/protocol.md b/protocol.md index 2ddef9d..ca7d2ca 100644 --- a/protocol.md +++ b/protocol.md @@ -568,6 +568,119 @@ Tus-Resumable: 1.0.0 Upload-Offset: 11 ``` +### Partial Checksum + +This extension allows Clients using the Checksum extension to verify and confirm the +integrity of the data from incomplete `PATCH` requests, thereby avoiding that data that +was received intact by the Server needs to be sent again. + +Servers that support this extension MUST add `partial-checksum` to the `Tus-Extension` header. + +If a Server receives an incomplete `PATCH` request with `Upload-Checksum` header, it +MAY store the partial data in a separate location rather than disposing of it. + +If the Server subsequently receives a `HEAD` request on the upload, it MAY provide range +and checksum information for the partially received data in `Upload-Partial-Checksum-Range` +and `Upload-Partial-Checksum` headers. + +A Client can then use the range and checksum information returned by the Server to +compute its own checksum for the range and compare it against the checksum returned +by the Server. + +If the checksums match, the Client MAY confirm the partial data's integrity by sending the +range and checksum information it wants to confirm in the `Upload-Metadata` header of +the next `PATCH` request, and resuming from the end of the confirmed range rather than the +`Upload-Offset` returned by the `HEAD` request. + +If the checksums don't match, the Client MUST continue the upload from the `Upload-Offset` +returned by the `HEAD` request. + +If the Server has kept the partial data and the Client confirms its integrity by sending identical +range and checksum information as `partial-upload-confirm-range` and +`partial-upload-confirm-checksum` in the `Upload-Metadata` header, +the Server MUST append the partial data to the upload and accept new data starting from the +new offset in the same `PATCH` request. + +If the Server has disposed of the partial data or if the Client sends range and checksum information +that doesn't match those of the Server for the partial data, the Server MUST dispose of the +partial data and MUST respond with a `409 Conflict` status. + +If the Server receives a `PATCH` request with an `Upload-Offset` other than the end of +the partial data's range, it SHOULD dispose of the partial data. + +Clients receiving a `409 Conflict` status in response SHOULD send a new `HEAD` request to the +Server to determine the upload's status before sending the next `PATCH` request. + +#### Headers + +##### Upload-Partial-Checksum-Range + +The `Upload-Partial-Checksum-Range` response header follows the format `[first included offset]-[last included offset]` +(f.ex. `0-499` for the first 500 bytes) and MUST describe the range of the partial data within +the upload. + +##### Upload-Partial-Checksum + +The `Upload-Partial-Checksum` response header uses the same format as the `Upload-Checksum` +header, but MUST contain the checksum of the partial data. + +##### Upload-Metadata + +Clients confirming partial data MUST send `partial-upload-confirm-range` and `partial-upload-confirm-checksum` as +metadata in the `Upload-Metadata` header. The formats for `partial-upload-confirm-range` and `partial-upload-confirm-checksum` +are identical to those of `Upload-Partial-Checksum-Range` and `Upload-Metadata` respectively. + +#### Examples + +**Request** + +``` +PATCH /files/17f44dbe1c4bace0e18ab850cf2b3a83 HTTP/1.1 +Content-Length: 11 +Upload-Offset: 0 +Tus-Resumable: 1.0.0 +Upload-Checksum: sha1 Kq5sNclPz7QV2+lfQIuc6R7oRu0= + +hello w[connection breaks] +``` + +**Request** + +``` +HEAD /files/17f44dbe1c4bace0e18ab850cf2b3a83 HTTP/1.1 +Tus-Resumable: 1.0.0 +``` + +**Response** + +``` +HTTP/1.1 200 OK +Upload-Offset: 0 +Upload-Partial-Checksum-Range: 0-7 +Upload-Partial-Checksum: sha1 V2uc6R7+lKq5sfQINclPz7QoRu0= +Tus-Resumable: 1.0.0 +``` + +**Request** + +``` +PATCH /files/17f44dbe1c4bace0e18ab850cf2b3a83 HTTP/1.1 +Upload-Offset: 7 +Upload-Metadata: partial-upload-confirm-range GFuLnBkZg==,partial-upload-confirm-checksum sfQININclV2uc6R7+lKq5sfQINclPz7QoRu0= +Content-Length: 4 +Tus-Resumable: 1.0.0 + +orld +``` + +**Response** + +``` +HTTP/1.1 204 No Content +Tus-Resumable: 1.0.0 +Upload-Offset: 11 +``` + ### Termination This extension defines a way for the Client to terminate completed and unfinished From e96856610802cdf0dabc1e66374ea58b65c88a29 Mon Sep 17 00:00:00 2001 From: Felix Schwarz Date: Tue, 23 Feb 2021 12:01:44 +0100 Subject: [PATCH 2/6] - clarify when partial data needs to be committed - add context to the example requests/responses --- protocol.md | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-) diff --git a/protocol.md b/protocol.md index ca7d2ca..3d74b7b 100644 --- a/protocol.md +++ b/protocol.md @@ -598,8 +598,9 @@ returned by the `HEAD` request. If the Server has kept the partial data and the Client confirms its integrity by sending identical range and checksum information as `partial-upload-confirm-range` and `partial-upload-confirm-checksum` in the `Upload-Metadata` header, -the Server MUST append the partial data to the upload and accept new data starting from the -new offset in the same `PATCH` request. +the Server MUST append the partial data to the upload. It MUST do this even if the `PATCH` +request that this information is part of is otherwise also not complete. The Server MUST also +accept new data starting from the new offset in the same `PATCH` request. If the Server has disposed of the partial data or if the Client sends range and checksum information that doesn't match those of the Server for the partial data, the Server MUST dispose of the @@ -632,6 +633,8 @@ are identical to those of `Upload-Partial-Checksum-Range` and `Upload-Metadata` #### Examples +The Client tries to append 11 bytes using a `PATCH` request: + **Request** ``` @@ -644,6 +647,8 @@ Upload-Checksum: sha1 Kq5sNclPz7QV2+lfQIuc6R7oRu0= hello w[connection breaks] ``` +The connection breaks down before the requests completes, so the Client sends a `HEAD` request to determine the current status: + **Request** ``` @@ -651,6 +656,8 @@ HEAD /files/17f44dbe1c4bace0e18ab850cf2b3a83 HTTP/1.1 Tus-Resumable: 1.0.0 ``` +The Server has kept the partially received data and adds range and checksum information about it to the response: + **Response** ``` @@ -661,6 +668,8 @@ Upload-Partial-Checksum: sha1 V2uc6R7+lKq5sfQINclPz7QoRu0= Tus-Resumable: 1.0.0 ``` +The Client verifies that the returned checksum is valid for the returned range, confirms it and resumes the upload from the end of the confirmed range: + **Request** ``` @@ -673,6 +682,8 @@ Tus-Resumable: 1.0.0 orld ``` +The request completes successfully and the Server responds with the new `Upload-Offset`: + **Response** ``` From 73e4f7e0d72456e8da87c8e80631af502ac14ca9 Mon Sep 17 00:00:00 2001 From: Felix Schwarz Date: Tue, 23 Feb 2021 12:18:51 +0100 Subject: [PATCH 3/6] - further clarifications --- protocol.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/protocol.md b/protocol.md index 3d74b7b..04697f5 100644 --- a/protocol.md +++ b/protocol.md @@ -599,8 +599,10 @@ If the Server has kept the partial data and the Client confirms its integrity by range and checksum information as `partial-upload-confirm-range` and `partial-upload-confirm-checksum` in the `Upload-Metadata` header, the Server MUST append the partial data to the upload. It MUST do this even if the `PATCH` -request that this information is part of is otherwise also not complete. The Server MUST also -accept new data starting from the new offset in the same `PATCH` request. +request that this information is part of is otherwise incomplete or still in transfer. This ensures +that multiple incomplete `PATCH` requests in a row are guaranteed to be possible without requiring +valid data to be re-sent. The Server MUST also accept an `Upload-Offset` starting from the end +of the partial data's range, in the same `PATCH` request. If the Server has disposed of the partial data or if the Client sends range and checksum information that doesn't match those of the Server for the partial data, the Server MUST dispose of the From c5ef4f3628546e2e2e008aeff28e5acda0f885f0 Mon Sep 17 00:00:00 2001 From: Felix Schwarz Date: Tue, 23 Feb 2021 12:19:30 +0100 Subject: [PATCH 4/6] - Clarifications --- protocol.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/protocol.md b/protocol.md index 04697f5..7d04030 100644 --- a/protocol.md +++ b/protocol.md @@ -602,7 +602,7 @@ the Server MUST append the partial data to the upload. It MUST do this even if t request that this information is part of is otherwise incomplete or still in transfer. This ensures that multiple incomplete `PATCH` requests in a row are guaranteed to be possible without requiring valid data to be re-sent. The Server MUST also accept an `Upload-Offset` starting from the end -of the partial data's range, in the same `PATCH` request. +of the confirmed partial data's range, in the same `PATCH` request. If the Server has disposed of the partial data or if the Client sends range and checksum information that doesn't match those of the Server for the partial data, the Server MUST dispose of the From 0892d1372f10ad77076e07c3db7fb037ba8aee17 Mon Sep 17 00:00:00 2001 From: Felix Schwarz Date: Fri, 26 Feb 2021 15:45:11 +0100 Subject: [PATCH 5/6] - replace usage of "Tus-Metadata" with bi-directional use of Upload-Partial-Checksum and Upload-Partial-Checksum-Range to simplify the implementation and specification - edit and partially rewrite for clarity and consistency with the rest of the specification - include POST requests into requests that can initiate Partial Checksum usage --- protocol.md | 79 +++++++++++++++++++++++++---------------------------- 1 file changed, 37 insertions(+), 42 deletions(-) diff --git a/protocol.md b/protocol.md index 7d04030..e926478 100644 --- a/protocol.md +++ b/protocol.md @@ -570,43 +570,42 @@ Upload-Offset: 11 ### Partial Checksum -This extension allows Clients using the Checksum extension to verify and confirm the -integrity of the data from incomplete `PATCH` requests, thereby avoiding that data that -was received intact by the Server needs to be sent again. +This extension allows Clients that use the Checksum extension to verify and confirm the +integrity of chunks that were not received in full by the Server - thereby avoiding that +intact data needs to be sent again. Servers that support this extension MUST add `partial-checksum` to the `Tus-Extension` header. -If a Server receives an incomplete `PATCH` request with `Upload-Checksum` header, it -MAY store the partial data in a separate location rather than disposing of it. +If a Server receives an incomplete `PATCH` or `POST` request with `Upload-Checksum` header, it +MAY store the incompletely received chunk's data (hereafter referred to as *partial data*) in +a separate location instead of discarding it. -If the Server subsequently receives a `HEAD` request on the upload, it MAY provide range -and checksum information for the partially received data in `Upload-Partial-Checksum-Range` -and `Upload-Partial-Checksum` headers. +If the Server receives a `HEAD` request on the upload thereafter, it MAY compute a checksum +on the partial data and then provide the result as well as the range of the partial data within +the upload through the `Upload-Partial-Checksum` and `Upload-Partial-Checksum-Range` +response headers. -A Client can then use the range and checksum information returned by the Server to -compute its own checksum for the range and compare it against the checksum returned -by the Server. +A Client can then use the range returned by the Server, compute its own checksum for the +same range and compare it against the checksum returned by the Server. -If the checksums match, the Client MAY confirm the partial data's integrity by sending the -range and checksum information it wants to confirm in the `Upload-Metadata` header of -the next `PATCH` request, and resuming from the end of the confirmed range rather than the -`Upload-Offset` returned by the `HEAD` request. +If the checksums are identical, the Client SHOULD confirm the partial data's integrity to the +Server by sending the checksum and range in the `Upload-Partial-Checksum` and +`Upload-Partial-Checksum-Range` headers of the next `PATCH` request. The `Upload-Offset` +header of the `PATCH` request MUST take the confirmed range into account and represent +the actual offset from where the upload should be resumed. If the checksums don't match, the Client MUST continue the upload from the `Upload-Offset` -returned by the `HEAD` request. +returned by the `HEAD` request, at which point the Server MAY discard any partial data it may +still be holding. -If the Server has kept the partial data and the Client confirms its integrity by sending identical -range and checksum information as `partial-upload-confirm-range` and -`partial-upload-confirm-checksum` in the `Upload-Metadata` header, -the Server MUST append the partial data to the upload. It MUST do this even if the `PATCH` -request that this information is part of is otherwise incomplete or still in transfer. This ensures -that multiple incomplete `PATCH` requests in a row are guaranteed to be possible without requiring -valid data to be re-sent. The Server MUST also accept an `Upload-Offset` starting from the end -of the confirmed partial data's range, in the same `PATCH` request. +If a Server receives `Upload-Partial-Checksum` and `Upload-Partial-Checksum-Range` +headers as part of a `PATCH` request and they match its own values for partial data stored +for the upload, the Server MUST append it to the upload and move its internal upload offset +forward accordingly, before processing the request any further. -If the Server has disposed of the partial data or if the Client sends range and checksum information -that doesn't match those of the Server for the partial data, the Server MUST dispose of the -partial data and MUST respond with a `409 Conflict` status. +If the Server has disposed of the partial data or if the Client sends range and checksum values +that do not match those of the Server for the partial data, the Server MUST dispose of the +partial data (if any) and MUST respond with a `409 Conflict` status. If the Server receives a `PATCH` request with an `Upload-Offset` other than the end of the partial data's range, it SHOULD dispose of the partial data. @@ -618,22 +617,17 @@ Server to determine the upload's status before sending the next `PATCH` request. ##### Upload-Partial-Checksum-Range -The `Upload-Partial-Checksum-Range` response header follows the format `[first included offset]-[last included offset]` -(f.ex. `0-499` for the first 500 bytes) and MUST describe the range of the partial data within -the upload. +The `Upload-Partial-Checksum-Range` header MUST describe the byte range of the partial +data within the upload, relative to the start of the upload, using this format: `[first byte offset]-[last byte offset]` + +For example `0-499` for the first 500 bytes, `500-999` for the second 500 bytes. ##### Upload-Partial-Checksum -The `Upload-Partial-Checksum` response header uses the same format as the `Upload-Checksum` +The `Upload-Partial-Checksum` header uses the same format as the `Upload-Checksum` header, but MUST contain the checksum of the partial data. -##### Upload-Metadata - -Clients confirming partial data MUST send `partial-upload-confirm-range` and `partial-upload-confirm-checksum` as -metadata in the `Upload-Metadata` header. The formats for `partial-upload-confirm-range` and `partial-upload-confirm-checksum` -are identical to those of `Upload-Partial-Checksum-Range` and `Upload-Metadata` respectively. - -#### Examples +#### Example The Client tries to append 11 bytes using a `PATCH` request: @@ -646,10 +640,10 @@ Upload-Offset: 0 Tus-Resumable: 1.0.0 Upload-Checksum: sha1 Kq5sNclPz7QV2+lfQIuc6R7oRu0= -hello w[connection breaks] +hello w[connection breaks down] ``` -The connection breaks down before the requests completes, so the Client sends a `HEAD` request to determine the current status: +Since the connection broke down before the request completed, the Client sends a `HEAD` request to determine from which offset it should resume: **Request** @@ -670,14 +664,15 @@ Upload-Partial-Checksum: sha1 V2uc6R7+lKq5sfQINclPz7QoRu0= Tus-Resumable: 1.0.0 ``` -The Client verifies that the returned checksum is valid for the returned range, confirms it and resumes the upload from the end of the confirmed range: +The Client verifies that the checksum provided by the Server is valid for the returned range, confirms they are identical and resumes the upload from the end of the confirmed range: **Request** ``` PATCH /files/17f44dbe1c4bace0e18ab850cf2b3a83 HTTP/1.1 Upload-Offset: 7 -Upload-Metadata: partial-upload-confirm-range GFuLnBkZg==,partial-upload-confirm-checksum sfQININclV2uc6R7+lKq5sfQINclPz7QoRu0= +Upload-Partial-Checksum-Range: 0-7 +Upload-Partial-Checksum: sha1 V2uc6R7+lKq5sfQINclPz7QoRu0= Content-Length: 4 Tus-Resumable: 1.0.0 From 32a812f1697abd57801bea4cca1094e0b3aa0e15 Mon Sep 17 00:00:00 2001 From: Felix Schwarz Date: Thu, 11 Mar 2021 16:23:27 +0100 Subject: [PATCH 6/6] - fixing typos (thanks @butonic and @smatsson) - massaging of several paragraphs to make them easier to read --- protocol.md | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/protocol.md b/protocol.md index e926478..7dd55e2 100644 --- a/protocol.md +++ b/protocol.md @@ -577,13 +577,13 @@ intact data needs to be sent again. Servers that support this extension MUST add `partial-checksum` to the `Tus-Extension` header. If a Server receives an incomplete `PATCH` or `POST` request with `Upload-Checksum` header, it -MAY store the incompletely received chunk's data (hereafter referred to as *partial data*) in +SHOULD store the incompletely received chunk's data (hereafter referred to as *partial data*) in a separate location instead of discarding it. -If the Server receives a `HEAD` request on the upload thereafter, it MAY compute a checksum -on the partial data and then provide the result as well as the range of the partial data within -the upload through the `Upload-Partial-Checksum` and `Upload-Partial-Checksum-Range` -response headers. +If the Server receives a `HEAD` request on the upload thereafter, it SHOULD compute a checksum +on the partial data and then return it, alongside the partial data's start and end offsets relative to +the beginning of the file, in the `Upload-Partial-Checksum` and `Upload-Partial-Checksum-Range` +response headers respectively. A Client can then use the range returned by the Server, compute its own checksum for the same range and compare it against the checksum returned by the Server. @@ -598,10 +598,10 @@ If the checksums don't match, the Client MUST continue the upload from the `Uplo returned by the `HEAD` request, at which point the Server MAY discard any partial data it may still be holding. -If a Server receives `Upload-Partial-Checksum` and `Upload-Partial-Checksum-Range` -headers as part of a `PATCH` request and they match its own values for partial data stored -for the upload, the Server MUST append it to the upload and move its internal upload offset -forward accordingly, before processing the request any further. +If a Server receives `Upload-Partial-Checksum` and `Upload-Partial-Checksum-Range` +headers as part of a `PATCH` request and they match the Server's computed results for its stored +partial data for the upload, the Server MUST append the partial data to the upload and move its +internal upload offset forward accordingly, before processing the request any further. If the Server has disposed of the partial data or if the Client sends range and checksum values that do not match those of the Server for the partial data, the Server MUST dispose of the @@ -659,7 +659,7 @@ The Server has kept the partially received data and adds range and checksum info ``` HTTP/1.1 200 OK Upload-Offset: 0 -Upload-Partial-Checksum-Range: 0-7 +Upload-Partial-Checksum-Range: 0-6 Upload-Partial-Checksum: sha1 V2uc6R7+lKq5sfQINclPz7QoRu0= Tus-Resumable: 1.0.0 ```