Envrionment
- Pynamodb:
6.0.2
- Boto3:
1.34.138
- BotoCore:
1.34.144
- python:
3.9
Description
- I encountered unexpected behavior regarding the
last_evaluated_key attribute of the ResultIterator returned by Model.sacn() when implementing manual pagination.
- It seems that the
last_evaluated_key attribute is correctly populated with the value from the Dynamodb response only after the iterator has been consumed(eg by calling list) or iterating through it in a for loop
- Accessing results.last_evaluated_key immediately after the scan() call returns, but before consuming the results iterator, consistently yielded None in my tests, even when the underlying DynamoDB response should have contained a non-null LastEvaluatedKey (indicating more pages are available). This prevents manual pagination logic that checks the key before processing the items.
Example
- Code order where pagination fails
results = MyModel.scan(filter_condition=filter_condition, limit=limit)
last_evaluated_key = results.last_evaluated_key
all_items.extend(list(results))
while last_evaluated_key: #<-- evaluted as none
scan_result = MyModel.scan(
last_evaluated_key=last_evaluated_key,
filter_condition=filter_condition,
limit=limit,
)
last_evaluated_key = results.last_evaluated_key
all_items.extend(list(scan_result))
- Code order where pagination works:
results = MyModel.scan(filter_condition=filter_condition, limit=limit)
all_items.extend(list(results))
last_evaluated_key = results.last_evaluated_key
while last_evaluated_key:
scan_result = MyModel.scan(
last_evaluated_key=last_evaluated_key,
filter_condition=filter_condition,
limit=limit,
)
all_items.extend(list(scan_result))
last_evaluated_key = scan_result.last_evaluated_key
