You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
for more details about the supported transformation blocks, and also the
339
-
[JMESPath custom functions]({{< relref "/integrate/redis-data-integration/reference/jmespath-custom-functions" >}}) reference. You can test your transformation logic using the [dry run]({{< relref "/integrate/redis-data-integration/reference/api-reference/#tag/secure/operation/job_dry_run_api_v1_pipelines_jobs_dry_run_post" >}}) feature in the API.
340
-
341
-
{{< note >}}If you set `row_format` to `full` under the `source` settings, you can access extra data from the
342
-
change record in the transformation:
343
-
- Use the `key` object to access the attributes of the key. For example, `key.id` will give you the value of the `id` column as long as it is part of the primary key.
344
-
- Use `before.<FIELD_NAME>` to get the value of a field *before* it was updated in the source database
345
-
- Use `after.<FIELD_NAME>` to get the value of a field *after* it was updated in the source database
346
-
- Use `after.<FIELD_NAME>` when adding new fields during transformations
347
-
348
-
See [Row Format]({{< relref "/integrate/redis-data-integration/data-pipelines/transform-examples/redis-row-format#full" >}}) for a more detailed explanation of the full format.
349
-
{{< /note >}}
350
-
351
-
- `output`: This is a mandatory section to specify the data structure(s) that
352
-
RDI will write to
353
-
the target along with the text pattern for the key(s) that will access it.
354
-
Note that you can map one record to more than one key in Redis or nest
for more information about nesting). You can add the following properties in the `output` section:
358
-
- `uses`: This must have the value `redis.write` to specify writing to a Redis data
359
-
structure. You can add more than one block of this type in the same job.
360
-
- `with`:
361
-
- `connection`: Connection name as defined in `config.yaml` (by default, the connection named `target` is used).
362
-
- `data_type`: Target data structure when writing data to Redis. The supported types are `hash`, `json`, `set`,
363
-
`sorted_set`, `stream` and `string`.
364
-
- `key`: This lets you override the default key for the data structure with custom logic:
365
-
- `expression`: Expression to generate the key.
366
-
- `language`: Expression language, which must be `jmespath` or `sql`.
367
-
- `expire`: Positive integer value indicating a number of seconds for the key to expire.
368
-
If you don't specify this property, the key will never expire.
369
-
370
-
{{< note >}}In a job file, the `transform` section is optional, but if you don't specify
371
-
a `transform`, you must specify custom key logic in `output.with.key`. You can include
372
-
both of these sections if you want both a custom transform and a custom key.{{< /note >}}
373
-
374
-
Another example below shows how you can rename the `fname` field to `first_name` in the table `emp`
375
-
using the
376
-
[`rename_field`]({{< relref "/integrate/redis-data-integration/reference/data-transformation/rename_field" >}}) block. It also demonstrates how you can set the key of this record instead of relying on
for more details about the supported transformation blocks, and also the
75
+
[JMESPath custom functions]({{< relref "/integrate/redis-data-integration/reference/jmespath-custom-functions" >}}) reference. You can test your transformation logic using the [dry run]({{< relref "/integrate/redis-data-integration/reference/api-reference/#tag/secure/operation/job_dry_run_api_v1_pipelines_jobs_dry_run_post" >}}) feature in the API.
76
+
77
+
{{< note >}}If you set `row_format` to `full` under the `source` settings, you can access extra data from the
78
+
change record in the transformation:
79
+
- Use the `key` object to access the attributes of the key. For example, `key.id` will give you the value of the `id` column as long as it is part of the primary key.
80
+
- Use `before.<FIELD_NAME>` to get the value of a field *before* it was updated in the source database
81
+
- Use `after.<FIELD_NAME>` to get the value of a field *after* it was updated in the source database
82
+
- Use `after.<FIELD_NAME>` when adding new fields during transformations
83
+
84
+
See [Row Format]({{< relref "/integrate/redis-data-integration/data-pipelines/transform-examples/redis-row-format#full" >}}) for a more detailed explanation of the full format.
85
+
{{< /note >}}
86
+
87
+
- `output`: This is a mandatory section to specify the data structure(s) that
88
+
RDI will write to
89
+
the target along with the text pattern for the key(s) that will access it.
90
+
Note that you can map one record to more than one key in Redis or nest
for more information about nesting). You can add the following properties in the `output` section:
94
+
- `uses`: This must have the value `redis.write` to specify writing to a Redis data
95
+
structure. You can add more than one block of this type in the same job.
96
+
- `with`:
97
+
- `connection`: Connection name as defined in `config.yaml` (by default, the connection named `target` is used).
98
+
- `data_type`: Target data structure when writing data to Redis. The supported types are `hash`, `json`, `set`,
99
+
`sorted_set`, `stream` and `string`.
100
+
- `key`: This lets you override the default key for the data structure with custom logic:
101
+
- `expression`: Expression to generate the key.
102
+
- `language`: Expression language, which must be `jmespath` or `sql`.
103
+
- `expire`: Positive integer value indicating a number of seconds for the key to expire.
104
+
If you don't specify this property, the key will never expire.
105
+
106
+
{{< note >}}In a job file, the `transform` section is optional, but if you don't specify
107
+
a `transform`, you must specify custom key logic in `output.with.key`. You can include
108
+
both of these sections if you want both a custom transform and a custom key.{{< /note >}}
109
+
110
+
Another example below shows how you can rename the `fname` field to `first_name` in the table `emp`
111
+
using the
112
+
[`rename_field`]({{< relref "/integrate/redis-data-integration/reference/data-transformation/rename_field" >}}) block. It also demonstrates how you can set the key of this record instead of relying on
0 commit comments