-
Notifications
You must be signed in to change notification settings - Fork 8
Open
Labels
Description
Description
Refactor the prepareMeasureSearch method to support pagination using the search_after parameter.
This intend to bypass Elastic Search 10K row limitations by using multiple requests
Tasks:
- Ensure proper sorting by measuredAt and _id.
- Return sorting criteria along with the search query.
- Update sendExport method to use search_after for pagination.
Example Code:
const sort = [
{ measuredAt: "asc" },
{ _id: "asc" } // Ensuring unique sorting by adding _id
];
return { digitalTwin, searchQuery, sort };Pros:
- Efficiently handles large datasets by paginating results.
- Avoids Elasticsearch limitations on result size.
Cons:
- Requires proper handling of search_after logic.
- Increases complexity of the search function.
Possible choices to made :
- Use PIT for data consistency but larger impact on ES that have to maintain data state for the search duration
- Do not use PIT as we export past data that should not be reordered between queries (with proper tie breaker on _ids)