Refactor Azure read model search to use Cosmos DB's native continuation token pagination #1587
Conversation
It refactors the search method in query helper of the Azure provider so it uses the continuation token instead of OFFSET and LIMIT for proper pagination.
PR Summary
|
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
|
/integration sha=df95c11 |
|
⌛ Integration tests are running... Check their status here 👈 |
|
❌ Oh no! Integration tests have failed |
|
/integration sha=7f4ef3b |
|
⌛ Integration tests are running... Check their status here 👈 |
|
❌ Oh no! Integration tests have failed |
Adds console.log messages for debugging event e2e integration tests
|
/integration sha=6b448ad |
|
⌛ Integration tests are running... Check their status here 👈 |
|
❌ Oh no! Integration tests have failed |
|
/integration sha=8b5abd15 |
|
⌛ Integration tests are running... Check their status here 👈 |
|
❌ Oh no! Integration tests have failed |
|
/integration sha=f1fdc38c |
|
⌛ Integration tests are running... Check their status here 👈 |
|
❌ Oh no! Integration tests have failed |
|
/integration sha=52e8593a |
|
⌛ Integration tests are running... Check their status here 👈 |
|
❌ Oh no! Integration tests have failed |
|
/integration sha=5565998a |
|
⌛ Integration tests are running... Check their status here 👈 |
|
❌ Oh no! Integration tests have failed |
|
/integration sha=4676cb04 |
|
⌛ Integration tests are running... Check their status here 👈 |
|
✅ Integration tests have finished successfully! |
|
❌ Oh no! Integration tests have failed |
|
/integration sha=01ac907e |
|
⌛ Integration tests are running... Check their status here 👈 |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
|
✅ Integration tests have finished successfully! |
There was a problem hiding this comment.
Pull Request Overview
This PR refactors the Azure read model search implementation to address pagination issues caused by unreliable OFFSET/LIMIT pagination in Cosmos DB. The implementation now uses Cosmos DB's native continuation token pagination for consistent results while maintaining backward compatibility with existing cursor formats.
- Replaced custom
OFFSET/LIMITpagination with Cosmos DB's continuation token system for reliable pagination - Added backward compatibility logic to handle legacy numeric cursors and DISTINCT queries that require
OFFSET/LIMITfallback - Updated integration tests to handle both continuation token and legacy cursor formats
Reviewed Changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
| query-helper.ts | Core refactoring of search method to implement continuation token pagination with fallback logic |
| read-models.integration.ts | Updated test expectations to handle both continuation token and legacy cursor formats |
| events.integration.ts | Added generic cursor type definition for flexible pagination handling |
| cosmos_db_pagination_2025-08-18-20-09.json | Added changelog entry documenting the pagination fix |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
|
/integration sha=1aff1c9c |
|
⌛ Integration tests are running... Check their status here 👈 |
|
❌ Oh no! Integration tests have failed |
|
⌛ Integration tests are running... Check their status here 👈 |
|
✅ Integration tests have finished successfully! |
Description
Currently, the implementation for read model search queries in the Azure provider uses custom pagination based on
OFFSETandLIMIT. This approach is causing records to either not appear in pages or be duplicated between pages, The root cause is that Cosmos DB does not guarantee consistent ordering for SQL queries without an explicitORDER BYclause, meaning that the same query can return results in different orders between page requests. This makesOFFSET-based pagination unreliable even without concurrent writes to the container. Additionally, the current cursor calculation logic simply increments by the limit size, which assumes no gaps in data and stable ordering. Using Cosmos DB's native continuation token pagination should correct this.Changes
search()method inquery-helper.tsto handle paginated queries using the continuation token.Checks
Updated documentation accordingly