-
Notifications
You must be signed in to change notification settings - Fork 15
s3 pagination failures #1177
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
s3 pagination failures #1177
Conversation
Signed-off-by: Evgenii Baidakov <evgenii@nspcc.io>
Change 1 This subpart of the ...
with allure.step("Check pagination with starting token"):
paginator = self.s3_client.get_paginator("list_object_versions")
next_token = None
all_versions = []
while True:
pagination_args = {"Bucket": bucket, "PaginationConfig": {"PageSize": 5}}
if next_token:
pagination_args["PaginationConfig"]["StartingToken"] = next_token
page_iterator = paginator.paginate(**pagination_args)
> page = next(page_iterator)
E TypeError: 'PageIterator' object is not an iterator with error
|
Change 2 I consider that there is a logical error in the We took max_keys (5 in the original code) and save it into We should update the test in the required way. Either change def test_s3_pagination_in_objects_versions_listing_via_boto3(self, prepare_versioned_objects: tuple):
bucket, objects_count = prepare_versioned_objects
with allure.step("Check basic pagination with MaxKeys"):
max_keys = 5 to def test_s3_pagination_in_objects_versions_listing_via_boto3(self, prepare_versioned_objects: tuple):
bucket, objects_count = prepare_versioned_objects
with allure.step("Check basic pagination with MaxKeys"):
max_keys = objects_count or another way, if the original assumption was different |
Change 3 inside
replace with
The reason is here. |
57674ff
to
d2aa83d
Compare
@smallhive |
@smallhive: all of the explanations belong to the commit message. That's exactly what it is for. |
d2aa83d
to
0acea6f
Compare
0acea6f
to
5d8bd0c
Compare
5d8bd0c
to
8c64adc
Compare
Reworked |
8c64adc
to
c2be36d
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Probably OK, we need more tests.
Closes #1175. ListObjectVersions method returns objects in the order that they were stored, returning the most recently stored object first. See details https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectVersions.html. To achieve this behaviour we extend `nextVersionIDMarker` with timestamp. It is used to filter out already processed objects. Signed-off-by: Evgenii Baidakov <evgenii@nspcc.io>
c2be36d
to
eaf71c9
Compare
Closes #1175
The tests themselves need to be updated as well