Added x-amz-expiration missing HTTP header in the response of object operations

[aayushchouhan09]

Describe the Problem

S3-compatible clients expect the x-amz-expiration response header when an object is subject to a lifecycle expiration rule. However, NooBaa was not setting this header in GET, PUT, or HEAD responses, even when valid expiration rules were configured.

Explain the Changes

1. Added set_expiration_header() function to include the x-amz-expiration header in GET, PUT, and HEAD object responses.
2. Implemented parse_expiration_header() to convert lifecycle expiration rules (days or date) into proper s3 style header format.

Below is the output for header of putObject, headObject and getObject (after the fix):

Issues: Fixed #xxx / Gap #xxx

1. Partially Fixed: NSFS | S3 | Lifecycle: bunch of expiration header tests failed #8554
2. Ceph-s3 tests are failing because in ceph tests they are calculating number of days without converting start_time to midnigt UTC (inside check_expiration_header() function) which is causing error and returning wrong no. of expiration days.
   ceph-s3 tests issue has been created for the same.

Testing Instructions:

1. Ceph-s3 tests are failing.
2. Added unit tests.

• Tests added

[aayushchouhan09]

PR is ready for review. For testing we can add ceph-s3 tests which are failing for missing x-amz-expiration header.

[shirady]

Comments and questions

[shirady]

Hi @achouhan09,

• I don't have prior knowledge about this header - if you can share what you tested in AWS it can be useful (I would also attach it in the PR description). I read this part in the blog to understand it quickly (link):

If you make a GET or a HEAD request on an object that has been scheduled for expiration, the response will include an x-amz-expiration header that includes this expiration date and the corresponding rule Id.

• I saw in the comment above you mentioned the failing Ceph tests - can you add them in this PR?
  You can remove them from the blacklist/pending list so they will run automatically in the CI.

• In AWS docs GetObject I found:

If the object expiration is configured (see PutBucketLifecycleConfiguration), the response includes this header. It includes the expiry-date and rule-id key-value pairs providing object expiration information. The value of the rule-id is URL-encoded.

Note - same comment in AWS docs HeadObject and also in AWS docs PutObject

Did you test in AWS the url-encoded part so we can understand what they do there?

I also saw this comment:

Note
Object expiration information is not returned in directory buckets and this header returns the value "NotImplemented" in all responses for directory buckets.

@romayalon @nadavMiz just want you to notice, I think it is fine to implement it in NSFS, but I would like you to think about it.

[aayushchouhan09]

Hi @achouhan09,

• I don't have prior knowledge about this header - if you can share what you tested in AWS it can be useful (I would also attach it in the PR description). I read this part in the blog to understand it quickly (link):

If you make a GET or a HEAD request on an object that has been scheduled for expiration, the response will include an x-amz-expiration header that includes this expiration date and the corresponding rule Id.

• I saw in the comment above you mentioned the failing Ceph tests - can you add them in this PR?
  You can remove them from the blacklist/pending list so they will run automatically in the CI.
• In AWS docs GetObject I found:

If the object expiration is configured (see PutBucketLifecycleConfiguration), the response includes this header. It includes the expiry-date and rule-id key-value pairs providing object expiration information. The value of the rule-id is URL-encoded.

Note - same comment in AWS docs HeadObject and also in AWS docs PutObject

Did you test in AWS the url-encoded part so we can understand what they do there?

I also saw this comment:

Note
Object expiration information is not returned in directory buckets and this header returns the value "NotImplemented" in all responses for directory buckets.

@romayalon @nadavMiz just want you to notice, I think it is fine to implement it in NSFS, but I would like you to think about it.

@shirady

• Updated PR description.
• Added ceph s3 tests - which are failing before the fix.
• Here, as per the aws documentation the rule-id should be url-encoded in x-amz-expiration but aws itself is not doing so, please check below screenshot:
  

cc @romayalon @nadavMiz

[nadavMiz]

since the documentation is not clear on this flag. I think it would also be good to check against aws if it exist for versioned objects as well. if it does it doesn't have to be in this PR, but at least we will open a gap for it