From b5419b6133f211a158a63173793a4e32edb8da1f Mon Sep 17 00:00:00 2001 From: "David W. Dougherty" Date: Tue, 1 Jul 2025 13:53:59 -0700 Subject: [PATCH 1/3] DEV: (RediSearch) doc expiration capabilities in Redis 8 --- content/commands/expire.md | 4 + content/commands/ft.aggregate.md | 1 + content/commands/ft.search.md | 1 + content/commands/hexpire.md | 4 + .../advanced-concepts/_index.md | 1 + .../advanced-concepts/expiration.md | 79 +++++++++++++++++++ 6 files changed, 90 insertions(+) create mode 100644 content/develop/ai/search-and-query/advanced-concepts/expiration.md diff --git a/content/commands/expire.md b/content/commands/expire.md index 3b3f663330..cde9c4251a 100644 --- a/content/commands/expire.md +++ b/content/commands/expire.md @@ -266,6 +266,10 @@ still take the full state of the expires existing in the dataset, so when a replica is elected to master it will be able to expire the keys independently, fully acting as a master. +### Redis Query Engine and expiration + +Starting with Redis 8, the Redis Query Engine has enhanced behavior when handling expiring keys. For detailed information about how [`FT.SEARCH`]({{< relref "/commands/ft.search" >}}) and [`FT.AGGREGATE`]({{< relref "/commands/ft.aggregate" >}}) commands interact with expiring keys, see [Key and field expiration behavior]({{< relref "/develop/ai/search-and-query/advanced-concepts/expiration" >}}). + ## Return information {{< multitabs id="expire-return-info" diff --git a/content/commands/ft.aggregate.md b/content/commands/ft.aggregate.md index ebd13fcb36..47aa2b4d28 100644 --- a/content/commands/ft.aggregate.md +++ b/content/commands/ft.aggregate.md @@ -496,5 +496,6 @@ Next, count GitHub events by user (actor), to produce the most active users. ## Related topics - [Aggregations]({{< relref "/develop/ai/search-and-query/advanced-concepts/aggregations" >}}) +- [Key and field expiration behavior]({{< relref "/develop/ai/search-and-query/advanced-concepts/expiration" >}}) - [RediSearch]({{< relref "/develop/ai/search-and-query" >}}) diff --git a/content/commands/ft.search.md b/content/commands/ft.search.md index 4961a64fc5..48d87fc42f 100644 --- a/content/commands/ft.search.md +++ b/content/commands/ft.search.md @@ -860,5 +860,6 @@ Query with `CONTAINS` operator: - [Extensions]({{< relref "/develop/ai/search-and-query/administration/extensions" >}}) - [Highlighting]({{< relref "/develop/ai/search-and-query/advanced-concepts/highlight" >}}) +- [Key and field expiration behavior]({{< relref "/develop/ai/search-and-query/advanced-concepts/expiration" >}}) - [Query syntax]({{< relref "/develop/ai/search-and-query/query/" >}}) - [RediSearch]({{< relref "/develop/ai/search-and-query/" >}}) diff --git a/content/commands/hexpire.md b/content/commands/hexpire.md index ff88e09161..77cc9f5830 100644 --- a/content/commands/hexpire.md +++ b/content/commands/hexpire.md @@ -113,6 +113,10 @@ You can call `HEXPIRE` using as argument a field that already has an existing TTL set. In this case, the time to live is _updated_ to the new value. +## Redis Query Engine and field expiration + +Starting with Redis 8, the Redis Query Engine has enhanced behavior when handling expiring hash fields. For detailed information about how [`FT.SEARCH`]({{< relref "/commands/ft.search" >}}) and [`FT.AGGREGATE`]({{< relref "/commands/ft.aggregate" >}}) commands interact with expiring hash fields, see [Key and field expiration behavior]({{< relref "/develop/ai/search-and-query/advanced-concepts/expiration" >}}). + ## Example ``` diff --git a/content/develop/ai/search-and-query/advanced-concepts/_index.md b/content/develop/ai/search-and-query/advanced-concepts/_index.md index 166217b290..3d755fc43d 100644 --- a/content/develop/ai/search-and-query/advanced-concepts/_index.md +++ b/content/develop/ai/search-and-query/advanced-concepts/_index.md @@ -42,6 +42,7 @@ Redis Open Source supports the following Redis Query Engine features. This artic * Numeric filters and ranges * Geo-filtering using Redis [geo commands]({{< relref "/commands/" >}}?group=geo) * [Vector search]({{< relref "/develop/ai/search-and-query/vectors" >}}) +* [Key and field expiration behavior]({{< relref "/develop/ai/search-and-query/advanced-concepts/expiration" >}}) ## Full-text search features diff --git a/content/develop/ai/search-and-query/advanced-concepts/expiration.md b/content/develop/ai/search-and-query/advanced-concepts/expiration.md new file mode 100644 index 0000000000..4aa37d7dd0 --- /dev/null +++ b/content/develop/ai/search-and-query/advanced-concepts/expiration.md @@ -0,0 +1,79 @@ +--- +categories: +- docs +- develop +- stack +- oss +- rs +- rc +- oss +- kubernetes +- clients +description: How the Redis Query Engine handles expiring keys and hash fields +linkTitle: Key and field expiration +title: Key and field expiration behavior +weight: 8 +--- + +The Redis Query Engine behavior with expiring keys and hash fields has been enhanced starting with Redis 8 to provide more consistent and predictable results. + +## Key expiration + +### Expiration times + +**Before Redis 8**: Expiration times were not taken into account when computing the result set. + +**Redis 8 and later**: The query engine prefers to return documents that are valid (not expired) at the time when the query or cursor read started. + +### Active expiration + +Active expiration can affect the number of results returned. For example, if a user requests 100 documents but 10 are actively expired during query execution, only 90 documents will be returned. + +**Note**: This behavior did not change in Redis 8 - active expiration has always affected result counts. + +### Passive expiration + +**Before Redis 8**: `nil` could have been returned as a document name in the result set for a query when a key was passively expired. + +**Redis 8 and later**: Only valid document names will be returned. Passively expired keys are filtered out from the result set. + +## Field expiration + +Field expiration was introduced in Redis 7.4 and provides fine-grained control over hash field lifecycles. + +### Expiration times + +**Before Redis 8**: Field expiration times were not taken into account when computing the result set. + +**Redis 8 and later**: The query engine prefers to return documents that are valid (fields not expired) at the time when the query or cursor read started. + +### Active expiration + +Similar to key expiration, active field expiration can affect the number of results returned. If fields that match query criteria are actively expired during execution, fewer results will be returned than requested. + +**Note**: This behavior did not change in Redis 8 - active expiration has always affected result counts. + +### Passive expiration + +**Before Redis 8**: The query engine would possibly return documents without fields that were passively expired, even if the expired field caused the document to match the query. This behavior depended to some degree on the `SORTABLE` keyword usage. + +**Redis 8 and later**: Documents will return with all fields that existed when the query or cursor read started. Passively expired fields are handled consistently. + +## Related commands + +The following commands are directly related to key and field expiration: + +- [`EXPIRE`]({{< relref "/commands/expire" >}}) - Set expiration time for keys +- [`HEXPIRE`]({{< relref "/commands/hexpire" >}}) - Set expiration time for hash fields +- [`FT.SEARCH`]({{< relref "/commands/ft.search" >}}) - Search queries affected by expiration behavior +- [`FT.AGGREGATE`]({{< relref "/commands/ft.aggregate" >}}) - Aggregation queries affected by expiration behavior + +## Best practices + +1. **Consistent timing**: Be aware that query results are based on the state at query start time, not when individual documents are processed. + +2. **Result count expectations**: When using expiring keys or fields, the actual number of results may be less than requested due to active expiration during query execution. + +3. **Field-level expiration**: Use hash field expiration (available since Redis 7.4) for fine-grained control over document field lifecycles without affecting the entire document. + +4. **Query planning**: Consider expiration patterns when designing queries, especially for time-sensitive applications where expired data should not appear in results. From 9a08256599718b7abf023d9f1a7e6af439883776 Mon Sep 17 00:00:00 2001 From: David Dougherty Date: Wed, 2 Jul 2025 06:35:00 -0700 Subject: [PATCH 2/3] Apply suggestions from code review Co-authored-by: andy-stark-redis <164213578+andy-stark-redis@users.noreply.github.com> --- .../ai/search-and-query/advanced-concepts/expiration.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/content/develop/ai/search-and-query/advanced-concepts/expiration.md b/content/develop/ai/search-and-query/advanced-concepts/expiration.md index 4aa37d7dd0..dff4b014df 100644 --- a/content/develop/ai/search-and-query/advanced-concepts/expiration.md +++ b/content/develop/ai/search-and-query/advanced-concepts/expiration.md @@ -33,7 +33,7 @@ Active expiration can affect the number of results returned. For example, if a u ### Passive expiration -**Before Redis 8**: `nil` could have been returned as a document name in the result set for a query when a key was passively expired. +**Before Redis 8**: A query could return `nil` as a document name in the result set for a key that was passively expired. **Redis 8 and later**: Only valid document names will be returned. Passively expired keys are filtered out from the result set. @@ -55,7 +55,7 @@ Similar to key expiration, active field expiration can affect the number of resu ### Passive expiration -**Before Redis 8**: The query engine would possibly return documents without fields that were passively expired, even if the expired field caused the document to match the query. This behavior depended to some degree on the `SORTABLE` keyword usage. +**Before Redis 8**: The query engine could return documents without fields that were passively expired, even if the expired field caused the document to match the query. This behavior depended to some degree on the `SORTABLE` keyword usage. **Redis 8 and later**: Documents will return with all fields that existed when the query or cursor read started. Passively expired fields are handled consistently. @@ -72,7 +72,7 @@ The following commands are directly related to key and field expiration: 1. **Consistent timing**: Be aware that query results are based on the state at query start time, not when individual documents are processed. -2. **Result count expectations**: When using expiring keys or fields, the actual number of results may be less than requested due to active expiration during query execution. +2. **Result count expectations**: Be aware that when you use expiring keys or fields, the actual number of results may be less than you expected due to active expiration during the execution of the query. 3. **Field-level expiration**: Use hash field expiration (available since Redis 7.4) for fine-grained control over document field lifecycles without affecting the entire document. From 4b47261a05bd0e44017f2fdda507a6b9352e1364 Mon Sep 17 00:00:00 2001 From: "David W. Dougherty" Date: Wed, 2 Jul 2025 06:44:52 -0700 Subject: [PATCH 3/3] Apply more suggestions from code review --- .../advanced-concepts/expiration.md | 24 +++++++++---------- 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/content/develop/ai/search-and-query/advanced-concepts/expiration.md b/content/develop/ai/search-and-query/advanced-concepts/expiration.md index dff4b014df..1e537d514d 100644 --- a/content/develop/ai/search-and-query/advanced-concepts/expiration.md +++ b/content/develop/ai/search-and-query/advanced-concepts/expiration.md @@ -23,7 +23,7 @@ The Redis Query Engine behavior with expiring keys and hash fields has been enha **Before Redis 8**: Expiration times were not taken into account when computing the result set. -**Redis 8 and later**: The query engine prefers to return documents that are valid (not expired) at the time when the query or cursor read started. +**Redis 8 and later**: The query engine returns only documents that are valid (not expired) at the time when the query or cursor read started. ### Active expiration @@ -45,7 +45,7 @@ Field expiration was introduced in Redis 7.4 and provides fine-grained control o **Before Redis 8**: Field expiration times were not taken into account when computing the result set. -**Redis 8 and later**: The query engine prefers to return documents that are valid (fields not expired) at the time when the query or cursor read started. +**Redis 8 and later**: The query engine returns only documents that are valid (fields not expired) at the time when the query or cursor read started. ### Active expiration @@ -59,6 +59,16 @@ Similar to key expiration, active field expiration can affect the number of resu **Redis 8 and later**: Documents will return with all fields that existed when the query or cursor read started. Passively expired fields are handled consistently. +## Best practices + +* **Consistent timing**: Be aware that query results are based on the state at query start time, not when individual documents are processed. + +* **Result count expectations**: Be aware that when you use expiring keys or fields, the actual number of results may be less than you expected due to active expiration during the execution of the query. + +* **Field-level expiration**: Use hash field expiration (available since Redis 7.4) for fine-grained control over document field lifecycles without affecting the entire document. + +* **Query planning**: Consider expiration patterns when designing queries, especially for time-sensitive applications where expired data should not appear in results. + ## Related commands The following commands are directly related to key and field expiration: @@ -67,13 +77,3 @@ The following commands are directly related to key and field expiration: - [`HEXPIRE`]({{< relref "/commands/hexpire" >}}) - Set expiration time for hash fields - [`FT.SEARCH`]({{< relref "/commands/ft.search" >}}) - Search queries affected by expiration behavior - [`FT.AGGREGATE`]({{< relref "/commands/ft.aggregate" >}}) - Aggregation queries affected by expiration behavior - -## Best practices - -1. **Consistent timing**: Be aware that query results are based on the state at query start time, not when individual documents are processed. - -2. **Result count expectations**: Be aware that when you use expiring keys or fields, the actual number of results may be less than you expected due to active expiration during the execution of the query. - -3. **Field-level expiration**: Use hash field expiration (available since Redis 7.4) for fine-grained control over document field lifecycles without affecting the entire document. - -4. **Query planning**: Consider expiration patterns when designing queries, especially for time-sensitive applications where expired data should not appear in results.