301/302 redirect HTTP codes should be followed

[drkspace]

If you try to access a bucket with an incorrect (but valid) region, you will get the following error (this was called with AWS_REGION=us-east-2 and the bucket is in us-east-1).

301  HTTP Error 301: Moved Permanently - Set s3auth.conf region to match bucket 'Region': bucket may not be in us-east-2  [PermanentRedirect] The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.

The redirect should be followed to get to the correct region and a warning message printed. There could also be an env variable/configuration option to follow/not to follow the redirect.

[le858]

[cpitchford]

#74

It seems the logic for the s3auth.conf file pulls the region (which in this context would be the bucket region) from s3auth.conf IF the file contains AccessKeyId. If not, it then tries $AWS_REGION, and lastly the EC2 metadata-service

When using an instance role, it would be useful if the Region value in the s3auth.conf were honoured when the credentials are pulled from the ec2 metadata.

It might not be advisable to set $AWS_REGION to a different region when installing packages since this might have implicated for the applications being installed, but it does allow apt-transport-s3 to reach buckets in different regions to the instance.

This PR will keep/use the Region setting in the s3auth.conf file even if the credentials are pulled from the environment or the metadata service.

[cpitchford]

#75 further updates by following the 301 response from S3.

The region and hostname is calculated and treats the Region from the config (s3auth.conf then $AWS_REGION then ec2 meta-data) as the default/starting region. If a 301 is returned, the new region is used; a new hostname is calculated and the new region is used in the AWSv4 signature.

The region is also cached per bucket.

This PR allows for multiple S3 buckets across multiple regions to be used as sources.