Create utility class URL builder and URL encoder. Fix a tricky URL path issue

[kingcrimsontianyu]

Background

Initial problem

There is currently an unsolved problem in libcurl, which somehow is mislabeled as merged/solved in curl/curl#13754. For AWS S3 that requires credentials, if an object key name contains =, libcurl will fail with an HTTP 403 response. This problem does not occur to public S3 objects. This can be reproduced using the curl program:

#!/usr/bin/env bash

# version: curl 8.15.0-DEV
curl_bin=<my_curl_program_loc>

# ..........S3 private..........
region=$(aws configure get region)
user_password=$(aws configure get aws_access_key_id):$(aws configure get aws_secret_access_key)
# curl can handle this. The object key name does not contain =
url="https://<private-bucket>.s3.<region>.amazonaws.com/witcher/2MiB.bin"
# curl cannot handle this. The object key name contains =
url="https://<private-bucket>.s3.<region>.amazonaws.com/witcher/key=value_2MiB.bin"

$curl_bin -s $url \
--aws-sigv4 "aws:amz:$region:s3" \
--user "$user_password" \
-o /dev/null -w "%{http_code}\n" -v

# ..........S3 public..........
# curl can handle both
url="https://<public-bucket>.s3.<region>.amazonaws.com/witcher/2MiB.bin"
url="https://<public-bucket>.s3.<region>.amazonaws.com/witcher/key=value_2MiB.bin"
$curl_bin -s $url \
-o /dev/null -w "%{http_code}\n" -v

Additional problem

It has been found that beyond = alone, other special characters such as !*'() in a private S3 object will also cause libcurl error. In addition, some characters such as + in a public S3 object will cause the same error.

This PR

This PR addresses this problem by handling special characters listed in the AWS object key naming guidelines, for both private and public S3 object names. The KvikIO-specific object key naming guidelines are added to the remote file documentation.

Specifically, this PR introduces utility classes UrlBuilder (to complement the existing UrlParser), which builds a URL according to the user-provided components, and UrlEncoder which uses a compile-time, percent-encoding lookup table to encode selected characters.

Closes #823

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[kingcrimsontianyu]

Updated documentation

[TomAugspurger]

Thanks @kingcrimsontianyu. The docs look good, and I confirmed that this fixes the issue we saw with = earlier.

I think as we discussed earlier, we weren't able to reproduce this issue with our moto-based tests because it handles authentication differently. So I think the tests in test_url.cpp are sufficient.

[kingcrimsontianyu]

Hmm. The URL cpp test failed on ARM + conda. Looking into this ...

[kingcrimsontianyu]

The test failure was caused by a misuse of string view, that resulted in UB. This has been fixed.
Also an oversight of mine, I forgot to include the control characters. They have now been added to the list of encoded characters.

[kingcrimsontianyu]

Redid the test that uses pylibcudf to read an exotically named S3 object "https://<bucket-private-or-public>.s3.<region>.amazonaws.com/witcher/!-_.*'()&$@=;:+%20,%3F/!-_.*'()&$@=;:+%20,%3F1_mil.parquet"

[kingcrimsontianyu]

/merge