Description
Describe the bug
The add_auth method of the CrtSigV4Auth and CrtSigV4AsymAuth signers fail to produce properly signed requests when the params of the request contain byte strings. This is because the signers perform the following:
for param, value in aws_request.params.items():
value = str(value)
array.append(f'{param}={value}')
The code str(value) when called on a byte string retains the b prefix. So, str(b'foo') returns: "b'foo'", which results in the following error message when making the request:
The request signature we calculated does not match the signature you provided.
This problem was discovered when I was making signed requests with these classes against an AWS OpenSearch cluster. When intercepting the requests and manually decoding the byte strings before signing, the signature became valid.
Relevant issue in the requests library: psf/requests#2238
Regression Issue
- Select this option if this issue appears to be a regression.
Expected Behavior
The CrtSigV4Auth and CrtSigV4AsymAuth should properly encode/decode the byte strings found in params in the add_auth method so valid signatures are created. This can be accomplished by changing value = str(value) to:
value = value.decode("utf-8") if isinstance(value, bytes) else str(value)
This will create valid signatures.
Current Behavior
Currently, add_auth simply does value = str(value) to all params resulting in invalid signatures when byte strings are present. Error:
The request signature we calculated does not match the signature you provided.
Reproduction Steps
It isn't feasible to provide a copy/paste solution since it's entirely contingent on being run against a service that you need to AWS sign against.
My use-case was when testing against an OpenSearch cluster with fine-grained access control enabled. Using the aforementioned signing classes, any requests with byte strings in the params would fail with the error:
The request signature we calculated does not match the signature you provided.
I created the following subclass of the Urllib3HttpConnection connection class that is used in the opensearchpy.OpenSearch client to remediate the issue:
import opensearchpy
from botocore.awsrequest import AWSRequest
from botocore.crt.auth import CrtSigV4Auth, CrtSigV4AsymAuth
class OSSignedUrllib3HttpConnection(opensearchpy.Urllib3HttpConnection):
def __init__(self, http_auth: CrtSigV4Auth | CrtSigV4AsymAuth, **kwargs):
self.aws_auth = http_auth
super().__init__(**kwargs)
def perform_request(
self,
method,
url,
params=None,
body=None,
timeout=None,
ignore=(),
headers=None,
):
for k, v in params.items():
if isinstance(v, bytes):
params[k] = v.decode("utf-8")
request = AWSRequest(method=method, headers=headers, url=self.host + url, data=body, params=params)
self.aws_auth.add_auth(request)
return super().perform_request(method, url, params=params, headers=request.headers, body=request.body, timeout=timeout, ignore=ignore)
You will see that the first lines of perform_request properly decode the byte strings. Remove that for-loop and the requests fail with invalid signatures. Note: Only when the request params contain a byte string.
Possible Solution
At these two lines:
- https://github.com/boto/botocore/blob/develop/botocore/crt/auth.py#L117
- https://github.com/boto/botocore/blob/develop/botocore/crt/auth.py#L315
Change value = str(value) to:
value = value.decode("utf-8") if isinstance(value, bytes) else str(value)
Additional Information/Context
No response
SDK version used
awscrt==0.26.1
botocore==1.37.32
Environment details (OS name and version, etc.)
Tested on multiple environments: Mac Apple M3 Max, EC2 cluster on AmazonLinux2023