Skip to content

[CSV Exporter] Update prepareMeasureSearch to Support search_after #349

@Juiced66

Description

@Juiced66

Description

Refactor the prepareMeasureSearch method to support pagination using the search_after parameter.
This intend to bypass Elastic Search 10K row limitations by using multiple requests

Tasks:

  • Ensure proper sorting by measuredAt and _id.
  • Return sorting criteria along with the search query.
  • Update sendExport method to use search_after for pagination.

Example Code:

const sort = [
  { measuredAt: "asc" },
  { _id: "asc" } // Ensuring unique sorting by adding _id
];

return { digitalTwin, searchQuery, sort };

Pros:

  • Efficiently handles large datasets by paginating results.
  • Avoids Elasticsearch limitations on result size.

Cons:

  • Requires proper handling of search_after logic.
  • Increases complexity of the search function.

Possible choices to made :

  • Use PIT for data consistency but larger impact on ES that have to maintain data state for the search duration
  • Do not use PIT as we export past data that should not be reordered between queries (with proper tie breaker on _ids)

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions