Skip to content

Commit 42d4306

Browse files
mich-elle-lunamich-elle-luna
authored
Merge pull request #1674 from ilianiliev-redis/RDSC-3621-format-epoch-to-date-string
RDSC-3621 format epoch to date string
2 parents dd730fd + df6f801 commit 42d4306

File tree

2 files changed

+200
-89
lines changed

2 files changed

+200
-89
lines changed

‎content/integrate/redis-data-integration/data-pipelines/transform-examples/formatting-date-and-time-values.md

Lines changed: 189 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,189 @@
1+
---
2+
Title: Formatting date and time values
3+
alwaysopen: false
4+
categories:
5+
- docs
6+
- integrate
7+
- rs
8+
- rdi
9+
description: null
10+
group: di
11+
linkTitle: Formatting date and time values
12+
summary: Redis Data Integration keeps Redis in sync with a primary database in near
13+
real time.
14+
type: integration
15+
weight: 40
16+
---
17+
18+
The way you format date and time values depends on the source database, the data type of the field, and how it is represented in the incoming record. Below are some examples for different databases and data types.
19+
20+
## Oracle
21+
22+
Oracle supports the following date and time data types:
23+
24+
- `DATE` - represented by Debezium as a 64-bit integer representing the milliseconds since epoch
25+
```yaml
26+
transform:
27+
- uses: add_field
28+
with:
29+
fields:
30+
- field: formatted_date
31+
language: sql
32+
# Date is stored as a Unix timestamp in milliseconds so you need to
33+
# divide it by 1000 to convert it to seconds.
34+
expression: STRFTIME('%Y-%m-%d %H:%M:%S', DATE / 1000, 'unixepoch')
35+
# Example: 1749047572000 is transformed to 2025-06-04 14:32:52
36+
```
37+
- `TIMESTAMP` - the value is represented by Debezium as a 64-bit integer and depends on the number of decimal places of precision of the column, representing fractions of a second. For example, if the column is defined as `TIMESTAMP(6)`, there are six decimal places and so the value is represented as microseconds since epoch (since there are 10^6 microseconds in each second).
38+
You can format it similarly to `DATE`, but you need to divide the value by the appropriate factor based on the precision.
39+
40+
- `TIMESTAMP WITH TIME ZONE` - the value is represented as a string containing the timestamp and time zone.
41+
42+
- `TIMESTAMP WITH LOCAL TIME ZONE` - the value is represented as a string containing the timestamp and local time zone.
43+
44+
SQLite supports both `TIMESTAMP WITH TIME ZONE` and `TIMESTAMP WITH LOCAL TIME ZONE`. You can format them using the `STRFTIME` function.
45+
46+
```yaml
47+
transform:
48+
- uses: add_field
49+
with:
50+
fields:
51+
- field: seconds_since_epoch
52+
language: sql
53+
# Convert the timestamp with local time zone to seconds since epoch.
54+
expression: STRFTIME('%s', TIMESTAMP_FIELD)
55+
56+
- field: date_from_timestamp
57+
language: sql
58+
# Convert the timestamp with local time zone to date and time.
59+
expression: STRFTIME('%Y-%m-%d %H:%M:%S', TIMESTAMP_FIELD)
60+
```
61+
62+
----
63+
64+
## SQL Server
65+
SQL Server supports the following date and time data types:
66+
67+
- `date` - represented by Debezium as number of days since epoch (1970-01-01). You can multiply the value by 86400 (the number of seconds in a day) to convert it to seconds since epoch and then use the `STRFTIME` or `DATE` functions to format it.
68+
```yaml
69+
transform:
70+
- uses: add_field
71+
with:
72+
fields:
73+
- field: with_default_date_format
74+
language: sql
75+
# Uses the default DATE format
76+
expression: DATE(event_date * 86400, 'unixepoch')
77+
78+
- field: with_custom_date_format
79+
language: sql
80+
# Uses the default DATE format
81+
expression: STRFTIME('%Y/%m/%d', event_date * 86400, 'unixepoch')
82+
```
83+
84+
- `datetime`, `smalldatetime` - represented by Debezium as number of milliseconds since epoch. Divide the value by 1000 to convert it to seconds since epoch and then use the `STRFTIME` function to format it.
85+
```yaml
86+
transform:
87+
- uses: add_field
88+
with:
89+
fields:
90+
- field: formatted_datetime
91+
language: sql
92+
expression: STRFTIME('%Y-%m-%d %H:%M:%S', event_datetime / 1000, 'unixepoch')
93+
```
94+
95+
- `datetime2` - similar to `datetime` but with higher precision. For `datetime2(0-3)`, the representation is the same as for `datetime`. For `datetime2(4-6)`, it is the number of microseconds since epoch. For `datetime2(7)`, it is the number of nanoseconds since epoch. To convert to another time unit, you can use the same approach as for `datetime` but you need to divide by 1000, 1000000 or 1000000000 depending on the precision.
96+
97+
- `time` - the number of milliseconds since midnight.
98+
```yaml
99+
transform:
100+
- uses: add_field
101+
with:
102+
fields:
103+
- field: formatted_time
104+
language: sql
105+
expression: TIME(event_time, 'unixepoch', 'utc')
106+
```
107+
108+
- `datetimeoffset` - represented as a timestamp with timezone information (for example, `2025-05-27T15:21:42.864Z` or `2025-01-02T14:45:30.123+05:00`).
109+
```yaml
110+
transform:
111+
- uses: add_field
112+
with:
113+
fields:
114+
- field: formatted_datetimeoffset
115+
language: sql
116+
expression: STRFTIME('%Y-%m-%d %H:%M:%S', event_datetimeoffset)
117+
```
118+
119+
120+
121+
122+
<!-- TODO [ilianiliev-redis]: Test and document the dynamic expressions for the rest of the supported databases - MySQL, PostgresSQL, MongoDB -->
123+
124+
125+
126+
----
127+
128+
## PostgreSQL
129+
130+
PostgreSQL supports the following date and time data types:
131+
132+
- `date` - represented by Debezium as number of days since epoch (1970-01-01). You can multiply the value by 86400 (the number of seconds in a day) to convert it to seconds since epoch and then use the `STRFTIME` or `DATE` functions to format it.
133+
```yaml
134+
transform:
135+
- uses: add_field
136+
with:
137+
fields:
138+
- field: with_default_date_format
139+
language: sql
140+
# Uses the default DATE format
141+
expression: DATE(event_date * 86400, 'unixepoch')
142+
```
143+
144+
- `time` - the time of microseconds since midnight.
145+
```yaml
146+
transform:
147+
- uses: add_field
148+
with:
149+
fields:
150+
- field: formatted_time
151+
language: sql
152+
# Divide by 1000000 to convert microseconds to seconds
153+
expression: TIME(event_time / 1000000, 'unixepoch', 'utc')
154+
```
155+
156+
- `time with time zone` - a string representation of the time with timezone information, where the timezone is GMT (for example, `07:15:00Z`).
157+
```yaml
158+
transform:
159+
- uses: add_field
160+
with:
161+
fields:
162+
- field: formatted_time_with_tz
163+
language: sql
164+
expression: STRFTIME('%H:%M:%S', event_time_with_time_zone)
165+
```
166+
167+
- `timestamp` - represented by Debezium as a 64-bit integer containing the microseconds since epoch. You can use the `STRFTIME` function to format it.
168+
```yaml
169+
transform:
170+
- uses: add_field
171+
with:
172+
fields:
173+
- field: formatted_timestamp
174+
language: sql
175+
# Divide by 1000000 to convert microseconds to seconds
176+
expression: STRFTIME('%Y-%m-%d %H:%M:%S', event_timestamp / 1000000, 'unixepoch')
177+
```
178+
179+
- `timestamp with time zone` - represented by Debezium as a string containing the timestamp with time zone information, where the timezone is GMT (for example, `2025-06-07T10:15:00.000000Z`).
180+
```yaml
181+
transform:
182+
- uses: add_field
183+
with:
184+
fields:
185+
- field: formatted_timestamp_with_tz
186+
language: sql
187+
# Divide by 1000000 to convert microseconds to seconds
188+
expression: STRFTIME('%Y-%m-%d %H:%M:%S', event_timestamp_with_time_zone)
189+
```

‎content/integrate/redis-data-integration/data-pipelines/transform-examples/redis-expiration-example.md

Lines changed: 11 additions & 89 deletions
Original file line numberDiff line numberDiff line change
@@ -53,112 +53,34 @@ output:
5353

5454
In some cases, you can also set the expiration time based on a field that contains a date, datetime, or timestamp value, but it depends on the source database and the data types it supports. See the examples below for your specific source database and data type.
5555

56-
### Oracle examples
56+
There are two main approaches you can use to set the expiration time based on a date, datetime, or timestamp field:
5757

58-
The transformation depends on the data type of the field in the source database:
58+
- For values representing an elapsed time since epoch start (in milliseconds, for example), you have to convert the value to seconds since epoch and then subtract the current time (also in seconds since epoch). The difference between the two is the time until expiration.
5959

60-
- `DATE` - represented by debezium as a 64-bit integer representing the milliseconds since epoch
61-
```yaml
62-
output:
63-
- uses: redis.write
64-
with:
65-
data_type: hash
66-
expire:
67-
# To set the expiration time to a date field, convert the value to seconds and subtract the current time in seconds since epoch
68-
expression: EXPIRES_DATE / 1000 - STRFTIME('%s', 'now')
69-
language: sql
70-
```
71-
- `TIMESTAMP` - the value is represented by Debezium as a 64-bit integer and depends on the number of decimal places of precision of the column, representing fractions of a second. For example, if the column is defined as `TIMESTAMP(6)`, there are six decimal places and so the value is represented as microseconds since epoch (since there are 10^6 microseconds in each second).
72-
```yaml
73-
output:
74-
- uses: redis.write
75-
with:
76-
data_type: hash
77-
expire:
78-
# To set the expiration time to a date field, convert the value to seconds (divider differs based on the fractional second precision) and subtract the current time in seconds since epoch. Example below is for 6 digits of precision.
79-
expression: EXPIRES_TIMESTAMP / 1000000 - STRFTIME('%s', 'now')
80-
language: sql
81-
```
82-
- `TIMESTAMP WITH TIME ZONE` - the value is represented as string representation of the timestamp with time zone information.
83-
- `TIMESTAMP WITH LOCAL TIME ZONE` - the value is represented as string representation of the timestamp with local time zone information.
84-
85-
For both `TIMESTAMP WITH TIME ZONE` and `TIMESTAMP WITH LOCAL TIME ZONE`, a two-step approach is needed. First, calculate the difference between the given time and now in seconds and then invert the value.
8660
```yaml
87-
transform:
88-
- uses: add_field
89-
with:
90-
fields:
91-
- field: expire_seconds
92-
language: jmespath
93-
expression: time_delta_seconds(EXPIRES_TS_TZ)
94-
output:
61+
output:
9562
- uses: redis.write
9663
with:
9764
data_type: hash
9865
expire:
99-
# `time_delta_seconds` Returns the number of seconds between a given dt and now.
100-
# A negative value means that the given dt is in the future, so we need to invert it.
101-
# A positive value means that the given dt is in the past, so set the expiration to -1 (expire immediately).
102-
expression: CASE WHEN expire_seconds < 0 THEN -expire_seconds ELSE -1 END
66+
# To set the expiration time to a date field, convert the value to
67+
# seconds (e.g. divide it by 1000 if the fields has milliseconds precision)
68+
# and subtract the current time in seconds since epoch.
69+
expression: EXPIRES_TIMESTAMP / 1000 - STRFTIME('%s', 'now')
10370
language: sql
10471
```
10572

106-
----
107-
108-
### SQL Server examples
109-
SQL Server supports the following date and time data types:
73+
- For values matching the subset of ISO 8601 supported by SQLite (for example, `2023-10-01T12:00:00`, `2023-10-01T12:00:00Z`, or `2025-06-05T13:40:14.784000+02:00`), you can use the `STRFTIME` function to convert the value to seconds since epoch and subtract the current time in seconds since epoch from it.
11074

111-
- `date` - represented in Debezium as number of days since epoch (1970-01-01). Please note that due to the lack of time information, this method is not very accurate.
112-
```yaml
113-
output:
114-
- uses: redis.write
115-
with:
116-
data_type: hash
117-
expire:
118-
# Calculate the number of seconds equivalent to the number of days and subtract the current time in seconds since epoch.
119-
expression: (event_date * 86400) - strftime('%s', 'now')
120-
language: sql
121-
```
122-
123-
- `datetime`, `smalldatetime` - represented in Debezium as number of milliseconds since epoch.
124-
```yaml
125-
output:
126-
- uses: redis.write
127-
with:
128-
data_type: hash
129-
expire:
130-
# Since event_datetime is in miliseconds, you must divide it by 1000 to convert it to seconds.
131-
expression: event_datetime / 1000 - strftime('%s', 'now')
132-
language: sql
133-
```
134-
- `datetime2` - similar to `datetime` but with higher precision. For `datetime2(0-3)` the representation is the same as for `datetime`. For `datetime2(4-6)` it is the number of microseconds since epoch. and for `datetime2(7)` it is the number of nanoseconds since epoch. You can use the same approach as for `datetime` but you need to divide by 1000, 1000000 or 1000000000 depending on the precision.
135-
136-
- `time` - the time of milliseconds since midnight.
137-
```yaml
138-
output:
139-
- uses: redis.write
140-
with:
141-
data_type: hash
142-
expire:
143-
# Convert the time to seconds and subtract the current time in seconds since midnight.
144-
expression: (event_time / 1000.0) -
145-
(
146-
CAST(strftime('%H', 'now') AS INTEGER) * 3600 +
147-
CAST(strftime('%M', 'now') AS INTEGER) * 60 +
148-
CAST(strftime('%S', 'now') AS INTEGER)
149-
)
150-
language: sql
151-
```
152-
- `datetimeoffset` - represented as a timestamp with timezone information, where the timezone is GMT
15375
```yaml
15476
output:
15577
- uses: redis.write
15678
with:
15779
data_type: hash
15880
expire:
159-
# Convert the time to seconds and subtract the current time in seconds since epoch.
160-
expression: strftime('%s', event_datetimeoffset) - strftime('%s', 'now')
16181
language: sql
82+
expression: STRFTIME('%s', EXPIRATION_TS) - STRFTIME('%s', 'now')
16283
```
16384

164-
<!-- TODO [ilianiliev-redis]: Test and document the dynamic expressions for the rest of the supported databases - MySQL, PostgresSQL, MongoDB -->
85+
For more examples of how to manipulate date and time values, see [Formatting date and time values]({{< relref "/integrate/redis-data-integration/data-pipelines/transform-examples/formatting-date-and-time-values/">}}).
86+

0 commit comments

Comments
 (0)