Checksum generation on multipart S3 uploads

[sophieilangolyon1-nhs]

This discussion relates to the FakeMultipart class for dealing with chunked S3 objects:

moto/moto/s3/models.py

Line 447 in c06ac47

def complete(

As of moto 5.1.2, per the changelog:

S3: get_object()/head_object() now return the 'x-amz-checksum-' header when appropriate

There seem to be two issues here when dealing with multipart (chunked) files and I wonder if anyone else has run into them.

The problems are:

• when a composite checksum is used, AWS adds a suffix -X onto the end of the checksum in the x-amz-checksum- header, so for example a composite checksum of UjZi/g== would result in the header x-amz-checksum-crc32: UjZi/g==-2 if there were two component parts. Moto does not add this suffix so the checksum presents as a full object checksum (see next point).
• moto generates composite checksums by default, which matches AWS behaviour, but AWS allows a ChecksumType flag to force FULL_OBJECT checksums - this feature is missing in moto, so only composite checksums (which have the previous problem) are supported

The upshot of this is that when using botocore and moto together, when reading the object back out of S3 botocore assumes the checksum is a FULL_OBJECT checksum (due to the missing -X suffix) and attempts to validate it against its own calculated full object checksum; this inevitably fails so the action to get the object fails with an exception.

I have attached some python code which can be run against AWS (with correct permissions) or against moto (with the 'mock' argument) to demonstrate the difference of behaviour. Note the two ExtraArgs options at line 27/28 which can be used to direct AWS towards composite or full object checksum generation.

reproduce-checksum-generation-problem.txt

[sophieil]

I am working on #8875 to address these issues