Skip to content

Commit 7d14d68

Browse files
ilianiliev-redisilianiliev-redis
committed
Moving time formatting examples to separate page
Updating examples of how to set keys TTL
1 parent 6dd8130 commit 7d14d68

File tree

2 files changed

+133
-89
lines changed

2 files changed

+133
-89
lines changed

‎content/integrate/redis-data-integration/data-pipelines/transform-examples/formatting-date-and-time-values.md

Lines changed: 122 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,122 @@
1+
---
2+
Title: Formatting date and time values
3+
alwaysopen: false
4+
categories:
5+
- docs
6+
- integrate
7+
- rs
8+
- rdi
9+
description: null
10+
group: di
11+
linkTitle: Formatting date and time values
12+
summary: Redis Data Integration keeps Redis in sync with a primary database in near
13+
real time.
14+
type: integration
15+
weight: 40
16+
---
17+
18+
How to format date and time values depends on the source database and the data type of the field, and how they are represented in the incoming record. Below are examples for different databases and data types.
19+
20+
## Oracle
21+
22+
Oracle supports the following date and time data types:
23+
24+
- `DATE` - represented by debezium as a 64-bit integer representing the milliseconds since epoch
25+
```yaml
26+
transform:
27+
- uses: add_field
28+
with:
29+
fields:
30+
- field: formatted_date
31+
language: sql
32+
# Date is stored as a Unix timestamp in milliseconds so you need to
33+
# divide it by 1000 to convert it to seconds.
34+
expression: STRFTIME('%Y-%m-%d %H:%M:%S', DATE / 1000, 'unixepoch')
35+
# Example: 1749047572000 is transformed to 2025-06-04 14:32:52
36+
```
37+
- `TIMESTAMP` - the value is represented by Debezium as a 64-bit integer and depends on the number of decimal places of precision of the column, representing fractions of a second. For example, if the column is defined as `TIMESTAMP(6)`, there are six decimal places and so the value is represented as microseconds since epoch (since there are 10^6 microseconds in each second).
38+
You can format it similarly to `DATE`, but you need to divide the value by the appropriate factor based on the precision.
39+
40+
- `TIMESTAMP WITH TIME ZONE` - the value is represented as string representation of the timestamp with time zone information.
41+
42+
- `TIMESTAMP WITH LOCAL TIME ZONE` - the value is represented as string representation of the timestamp with local time zone information.
43+
44+
Both `TIMESTAMP WITH TIME ZONE` and `TIMESTAMP WITH LOCAL TIME ZONE` are supported by SQLite and can be formatted using the `STRFTIME` function.
45+
46+
```yaml
47+
transform:
48+
- uses: add_field
49+
with:
50+
fields:
51+
- field: seconds_since_epoch
52+
language: sql
53+
# Convert the timestamp with local time zone to seconds since epoch.
54+
expression: STRFTIME('%s', TIMESTAMP_FIELD)
55+
56+
- field: date_from_timestamp
57+
language: sql
58+
# Convert the timestamp with local time zone to date and time.
59+
expression: STRFTIME('%Y-%m-%d %H:%M:%S', TIMESTAMP_FIELD)
60+
```
61+
62+
----
63+
64+
## SQL Server
65+
SQL Server supports the following date and time data types:
66+
67+
- `date` - represented by Debezium as number of days since epoch (1970-01-01). You can multiply the value by 86400 (the number of seconds in a day) to convert it to seconds since epoch and then use the `STRFTIME` or `DATE` functions to format it.
68+
```yaml
69+
transform:
70+
- uses: add_field
71+
with:
72+
fields:
73+
- field: with_default_date_format
74+
language: sql
75+
# Uses the default DATE format
76+
expression: DATE(event_date * 86400, 'unixepoch')
77+
78+
- field: with_custom_date_format
79+
language: sql
80+
# Uses the default DATE format
81+
expression: STRFTIME('%Y/%m/%d', event_date * 86400, 'unixepoch')
82+
```
83+
84+
- `datetime`, `smalldatetime` - represented by Debezium as number of milliseconds since epoch. You have to divide the value by 1000 to convert it to seconds since epoch and then use the `STRFTIME` function to format it.
85+
```yaml
86+
transform:
87+
- uses: add_field
88+
with:
89+
fields:
90+
- field: formatted_datetime
91+
language: sql
92+
expression: STRFTIME('%Y-%m-%d %H:%M:%S', event_datetime / 1000, 'unixepoch')
93+
```
94+
95+
- `datetime2` - similar to `datetime` but with higher precision. For `datetime2(0-3)` the representation is the same as for `datetime`. For `datetime2(4-6)` it is the number of microseconds since epoch. and for `datetime2(7)` it is the number of nanoseconds since epoch. You can use the same approach as for `datetime` but you need to divide by 1000, 1000000 or 1000000000 depending on the precision.
96+
97+
- `time` - the time of milliseconds since midnight.
98+
```yaml
99+
transform:
100+
- uses: add_field
101+
with:
102+
fields:
103+
- field: formatted_time
104+
language: sql
105+
expression: TIME(event_time, 'unixepoch', 'utc')
106+
```
107+
108+
- `datetimeoffset` - represented as a timestamp with timezone information e.g. `2025-05-27T15:21:42.864Z` and `2025-01-02T14:45:30.123+05:00`.
109+
```yaml
110+
transform:
111+
- uses: add_field
112+
with:
113+
fields:
114+
- field: formatted_datetimeoffset
115+
language: sql
116+
expression: STRFTIME('%Y-%m-%d %H:%M:%S', event_datetimeoffset)
117+
```
118+
119+
120+
121+
122+
<!-- TODO [ilianiliev-redis]: Test and document the dynamic expressions for the rest of the supported databases - MySQL, PostgresSQL, MongoDB -->

‎content/integrate/redis-data-integration/data-pipelines/transform-examples/redis-expiration-example.md

Lines changed: 11 additions & 89 deletions
Original file line numberDiff line numberDiff line change
@@ -53,112 +53,34 @@ output:
5353

5454
In some cases, you can also set the expiration time based on a field that contains a date, datetime, or timestamp value, but it depends on the source database and the data types it supports. See the examples below for your specific source database and data type.
5555

56-
### Oracle examples
56+
There are two main approaches you can use to set the expiration time based on a date, datetime, or timestamp field:
5757

58-
The transformation depends on the data type of the field in the source database:
58+
- For values representing a passed amount of time e.g. milliseconds since epoch start, you have to convert the value to seconds since epoch and subtracting the current time in seconds since epoch from it.
5959

60-
- `DATE` - represented by debezium as a 64-bit integer representing the milliseconds since epoch
61-
```yaml
62-
output:
63-
- uses: redis.write
64-
with:
65-
data_type: hash
66-
expire:
67-
# To set the expiration time to a date field, convert the value to seconds and subtract the current time in seconds since epoch
68-
expression: EXPIRES_DATE / 1000 - STRFTIME('%s', 'now')
69-
language: sql
70-
```
71-
- `TIMESTAMP` - the value is represented by Debezium as a 64-bit integer and depends on the number of decimal places of precision of the column, representing fractions of a second. For example, if the column is defined as `TIMESTAMP(6)`, there are six decimal places and so the value is represented as microseconds since epoch (since there are 10^6 microseconds in each second).
72-
```yaml
73-
output:
74-
- uses: redis.write
75-
with:
76-
data_type: hash
77-
expire:
78-
# To set the expiration time to a date field, convert the value to seconds (divider differs based on the fractional second precision) and subtract the current time in seconds since epoch. Example below is for 6 digits of precision.
79-
expression: EXPIRES_TIMESTAMP / 1000000 - STRFTIME('%s', 'now')
80-
language: sql
81-
```
82-
- `TIMESTAMP WITH TIME ZONE` - the value is represented as string representation of the timestamp with time zone information.
83-
- `TIMESTAMP WITH LOCAL TIME ZONE` - the value is represented as string representation of the timestamp with local time zone information.
84-
85-
For both `TIMESTAMP WITH TIME ZONE` and `TIMESTAMP WITH LOCAL TIME ZONE`, a two-step approach is needed. First, calculate the difference between the given time and now in seconds and then invert the value.
8660
```yaml
87-
transform:
88-
- uses: add_field
89-
with:
90-
fields:
91-
- field: expire_seconds
92-
language: jmespath
93-
expression: time_delta_seconds(EXPIRES_TS_TZ)
94-
output:
61+
output:
9562
- uses: redis.write
9663
with:
9764
data_type: hash
9865
expire:
99-
# `time_delta_seconds` Returns the number of seconds between a given dt and now.
100-
# A negative value means that the given dt is in the future, so we need to invert it.
101-
# A positive value means that the given dt is in the past, so set the expiration to -1 (expire immediately).
102-
expression: CASE WHEN expire_seconds < 0 THEN -expire_seconds ELSE -1 END
66+
# To set the expiration time to a date field, convert the value to
67+
# seconds (e.g. divide it by 1000 if the fields has milliseconds precision)
68+
# and subtract the current time in seconds since epoch.
69+
expression: EXPIRES_TIMESTAMP / 1000 - STRFTIME('%s', 'now')
10370
language: sql
10471
```
10572

106-
----
107-
108-
### SQL Server examples
109-
SQL Server supports the following date and time data types:
73+
- For values matching the subset of ISO 8601 supported by SQLite e.g. `2023-10-01T12:00:00`, `2023-10-01T12:00:00Z` or `2025-06-05T13:40:14.784000+02:00` you can use the `STRFTIME` function to convert the value to seconds since epoch and subtract the current time in seconds since epoch from it.
11074

111-
- `date` - represented in Debezium as number of days since epoch (1970-01-01). Please note that due to the lack of time information, this method is not very accurate.
112-
```yaml
113-
output:
114-
- uses: redis.write
115-
with:
116-
data_type: hash
117-
expire:
118-
# Calculate the number of seconds equivalent to the number of days and subtract the current time in seconds since epoch.
119-
expression: (event_date * 86400) - strftime('%s', 'now')
120-
language: sql
121-
```
122-
123-
- `datetime`, `smalldatetime` - represented in Debezium as number of milliseconds since epoch.
124-
```yaml
125-
output:
126-
- uses: redis.write
127-
with:
128-
data_type: hash
129-
expire:
130-
# Since event_datetime is in miliseconds, you must divide it by 1000 to convert it to seconds.
131-
expression: event_datetime / 1000 - strftime('%s', 'now')
132-
language: sql
133-
```
134-
- `datetime2` - similar to `datetime` but with higher precision. For `datetime2(0-3)` the representation is the same as for `datetime`. For `datetime2(4-6)` it is the number of microseconds since epoch. and for `datetime2(7)` it is the number of nanoseconds since epoch. You can use the same approach as for `datetime` but you need to divide by 1000, 1000000 or 1000000000 depending on the precision.
135-
136-
- `time` - the time of milliseconds since midnight.
137-
```yaml
138-
output:
139-
- uses: redis.write
140-
with:
141-
data_type: hash
142-
expire:
143-
# Convert the time to seconds and subtract the current time in seconds since midnight.
144-
expression: (event_time / 1000.0) -
145-
(
146-
CAST(strftime('%H', 'now') AS INTEGER) * 3600 +
147-
CAST(strftime('%M', 'now') AS INTEGER) * 60 +
148-
CAST(strftime('%S', 'now') AS INTEGER)
149-
)
150-
language: sql
151-
```
152-
- `datetimeoffset` - represented as a timestamp with timezone information, where the timezone is GMT
15375
```yaml
15476
output:
15577
- uses: redis.write
15678
with:
15779
data_type: hash
15880
expire:
159-
# Convert the time to seconds and subtract the current time in seconds since epoch.
160-
expression: strftime('%s', event_datetimeoffset) - strftime('%s', 'now')
16181
language: sql
82+
expression: STRFTIME('%s', EXPIRATION_TS) - STRFTIME('%s', 'now')
16283
```
16384

164-
<!-- TODO [ilianiliev-redis]: Test and document the dynamic expressions for the rest of the supported databases - MySQL, PostgresSQL, MongoDB -->
85+
For more examples on how to manipulate date and time values, see the [Formatting date and time values]({{< relref "/integrate/redis-data-integration/data-pipelines/transform-examples/formatting-date-and-time-values/">}}) page.
86+

0 commit comments

Comments
 (0)