Import Binary Data as multipart or base64 inline

[travisgosselin]

It is difficult to mix binary data with RESTful resources. For example, when you want a RESTful endpoint that creates a new object, but that item is a combination of metadata and binary data, then where do the binary data and metadata get included?

Option 1: Headers & Content-Type (octet-stream)

In this scenario, we use the "Content-Type" as intended to tell the request that the entire incoming body is of type application/octet-stream for incoming binary data like an image, or xls spreadsheet. The metadata about that object is included in headers.

POST /v1/spreadsheets
CONTENT-TYPE: application/octet-stream
SPS-File-Name: my-file.xls
SPS-Relationship: xxx-xxx-xxxxx

Generally speaking, this is not ideal headers are a bit more unfriendly and limited in their flexibility/discoverability.

Option 2: Multipart/FormData

POST /v1/spreadsheets
CONTENT-TYPE: multipart/form-data
FormData:      file: my-file.xls
FormData:     relationship: xxx-xxx-xxxxx

This is more flexible than the first option enabling additional form data to be added as part of the payload/form-data directly.

Option 3: Body + Base64 Binary Inline

In this scenario, we make our life simple by handling standard JSON in our restful resource which is very consistent. The binary data is included in a base64 string value representing the spreadsheet that must be converted (both by client and backend).

POST /v1/spreadsheets
CONTENT-TYPE: application/json
{
   "fileName": "my-file.xls",
  "relationship": "xxx-xxx-xxxxx",
  "file": "g4FtieJfU51Af2Cr1+h5WTTAnpdYREU3Eg=="
}

Some implications: https://stackoverflow.com/questions/65350640/to-upload-a-file-what-are-the-pros-and-cons-of-sending-base64-in-post-body-vs-mu

Option 4: Separate Metadata and Binary Endpoints

In this scenario, it is a hybrid between the above approaches. There is too much metadata to be specified as all headers or can even be confusing as purely form-data. Additionally, using multipart/form-data for partial upload is not an option with base64 encoding. The overhead in compute and size of base64 encoding makes option 3 very specific for certain circumstances.

Create a spreadsheet object that we can then upload other types of content too:

POST /v1/spreadsheets
CONTENT-TYPE: application/json
{
   "fileName": "my-file.xls",
   "relationship": "xxx-xxx-xxxxx"
}

This endpoint provides back a unique resource id we can replace for the content

PUT /v1/spreadsheets/{id}/content
CONTENT-TYPE: multipart/form-data
FormData:        fileSize: 54                             # include file-specific form data if needed.

This hybrid pattern while two calls, does enable the best of both worlds:

• Flexible to evolve the resource with standard RESTful endpoints/verbs for actions on the content.
• Provides a complete range of an approach for partial uploads not re-inventing or incurring the overhead of non-standard base64 encoding.
• Allows the incorporation of other sub-resource types for the resource (i.e. content could be "image1" and "image2" endpoints).
• Enables off-loading the download of binary files to potentially non-restful resources.
• Location of binary data downloads could be easily made available in linked fields for the resource:

GET /v1/spreadsheets/{id}
CONTENT-TYPE: application/json
{
   "fileName": "my-file.xls",
   "relationship": "xxx-xxx-xxxxx",
  "fileUrl": "https://thing.com/v1/spreadsheets/{id}/content"
}

Some comparisons:

• Shopify allows for product image uploads via Base64 - https://shopify.dev/docs/api/admin-rest/2023-01/resources/product-image#post-products-product-id-images
• Stripe uses multipart/form-data for creation - https://stripe.com/docs/api/files/create?lang=curl

In simpler scenarios combining JSON and multipart endpoints works well:

POST /v1/spreadsheets
CONTENT-TYPE: multipart/form-data
FormData:        fileName="my-file.xls"

POST /v1/spreadsheets
CONTENT-TYPE: application/json
{
    "fileName": "my-file.xls"
}

Using RESTful resources with either application/json or multipart/form-data as originally intended and extending to sub-resources if complex enough feels like a natural approach based on the available primitives. Thoughts?

[jessecoyle]

Multipart formdata is very attractive, but not well supported by clients. Often special extensions are needed which aren't very widely used or in some cases the format has to be handled manually.

One consideration is payload size. This can make encodings like base64 unmanageable. A single binary format can make it easier to support range requests for capable clients to resume interrupted downloads. This can also make it much easier for browsers to open the content directly when that is desired.

One thought that is under consideration for parcel service is whether and how to support resuming uploads. One option is to support uploading in chunks. This is supported by s3.

[travisgosselin]

Not entirely sure if your offering an opinion above or just adding more info. If I had to interpret it, I'd say you are suggesting that all options are relevant and we should use different options stated above depending on the situation and the audience?

Are you able to provide an example of the special extensions your referring to as well?