Skip to content

Commit 77149f2

Browse files
andy-stark-redisandy-stark-redis
authored
Merge pull request #829 from redis/DOC-4510
DOC-4510 RDI add_field and remove_field job file examples
2 parents 8379b1b + e224d38 commit 77149f2

File tree

2 files changed

+342
-0
lines changed

2 files changed

+342
-0
lines changed

‎content/integrate/redis-data-integration/data-pipelines/transform-examples/redis-add-field-example.md

Lines changed: 176 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,176 @@
1+
---
2+
Title: Add new fields to a key
3+
alwaysopen: false
4+
categories:
5+
- docs
6+
- integrate
7+
- rs
8+
- rdi
9+
description: null
10+
group: di
11+
linkTitle: Add new fields
12+
summary: Redis Data Integration keeps Redis in sync with a primary database in near
13+
real time.
14+
type: integration
15+
weight: 40
16+
---
17+
18+
By default, RDI adds fields to
19+
[hash]({{< relref "/develop/data-types/hashes" >}}) or
20+
[JSON]({{< relref "/develop/data-types/json" >}}) objects in the target
21+
database that match the columns of the source table.
22+
The examples below show how to add extra fields to the target data with the
23+
[`add_field`]({{< relref "/integrate/redis-data-integration/reference/data-transformation/add_field" >}}) transformation.
24+
25+
## Add a single field
26+
27+
The first example adds a single field to the data.
28+
The `source` section selects the `customer` table of the
29+
[`chinook`](https://github.com/Redislabs-Solution-Architects/rdi-quickstart-postgres)
30+
database (the optional `db` value here corresponds to the
31+
`sources.<source-name>.connection.database` value defined in
32+
[`config.yaml`]({{< relref "/integrate/redis-data-integration/data-pipelines/data-pipelines#the-configyaml-file" >}})).
33+
34+
In the `transform` section, the `add_field` transformation adds an extra field called `localphone`
35+
to the object, which is created by removing the country and area code from the `phone`
36+
field with the
37+
[JMESPath]({{< relref "/integrate/redis-data-integration/reference/jmespath-custom-functions" >}}) function `regex_replace()`.
38+
You can also specify `sql` as the `language` if you prefer to create the new
39+
field with an [SQL](https://en.wikipedia.org/wiki/SQL) expression.
40+
41+
The `output` section specifies `hash` as the `data_type` to write to the target, which
42+
overrides the default setting of `target_data_type` defined in `config.yaml`. Also, the
43+
`output.with.key` section specifies a custom key format of the form `cust:<id>` where
44+
the `id` part is generated by the `uuid()` function.
45+
46+
The full example is shown below:
47+
48+
```yaml
49+
source:
50+
db: chinook
51+
table: customer
52+
transform:
53+
- uses: add_field
54+
with:
55+
expression: regex_replace(phone, '\+[0-9]+ (\([0-9]+\) )?', '')
56+
field: localphone
57+
language: jmespath
58+
output:
59+
- uses: redis.write
60+
with:
61+
connection: target
62+
data_type: hash
63+
key:
64+
expression: concat(['cust:', uuid()])
65+
language: jmespath
66+
```
67+
68+
If you queried the generated target data from the default transformation
69+
using [`redis-cli`]({{< relref "/develop/connect/cli" >}}), you would
70+
see something like the following:
71+
72+
```
73+
1) "customerid"
74+
2) "27"
75+
3) "firstname"
76+
4) "Patrick"
77+
5) "lastname"
78+
6) "Gray"
79+
.
80+
.
81+
17) "phone"
82+
18) "+1 (520) 622-4200"
83+
.
84+
.
85+
```
86+
87+
Using the job file above, the data also includes the new `localphone` field:
88+
89+
```
90+
1) "customerid"
91+
2) "27"
92+
3) "firstname"
93+
4) "Patrick"
94+
5) "lastname"
95+
6) "Gray"
96+
.
97+
.
98+
23) "localphone"
99+
24) "622-4200"
100+
```
101+
102+
## Add multiple fields
103+
104+
The `add_field` transformation can also add multiple fields at the same time
105+
if you specify them under a `fields` subsection. The example below adds two
106+
fields to the `track` objects. The first new field, `seconds`, is created using a SQL
107+
expression to calculate the duration of the track in seconds from the
108+
`milliseconds` field.
109+
The second new field, `composerlist`, adds a JSON array using the `split()` function
110+
to split the `composer` string field wherever it contains a comma.
111+
112+
```yaml
113+
source:
114+
db: chinook
115+
table: track
116+
transform:
117+
- uses: add_field
118+
with:
119+
fields:
120+
- expression: floor(milliseconds / 1000)
121+
field: seconds
122+
language: sql
123+
- expression: split(composer)
124+
field: composerlist
125+
language: jmespath
126+
output:
127+
- uses: redis.write
128+
with:
129+
connection: target
130+
data_type: json
131+
key:
132+
expression: concat(['track:', trackid])
133+
language: jmespath
134+
```
135+
136+
You can query the target database to see the new fields in
137+
the JSON object:
138+
139+
```bash
140+
> JSON.GET track:1 $
141+
142+
"[{\"trackid\":1,\"name\":\"For Those About To Rock (We Salute You)\",\"albumid\":1,\"mediatypeid\":1,\"genreid\":1,\"composer\":\"Angus Young, Malcolm Young, Brian Johnson\",\"milliseconds\":343719,\"bytes\":11170334,\"unitprice\":\"0.99\",\"seconds\":343,\"composerlist\":[\"Angus Young\",\" Malcolm Young\",\" Brian Johnson\"]}]"
143+
```
144+
145+
## Using `add_field` with `remove_field`
146+
147+
You can use the `add_field` and
148+
[`remove_field`]({{< relref "/integrate/redis-data-integration/data-pipelines/transform-examples/redis-remove-field-example" >}})
149+
transformations together to completely replace fields from the source. For example,
150+
if you add a new `fullname` field, you might not need the separate `firstname` and
151+
`lastname` fields. You can remove them with a job file like the following:
152+
153+
```yaml
154+
source:
155+
db: chinook
156+
table: customer
157+
transform:
158+
- uses: add_field
159+
with:
160+
expression: concat(firstname, ' ', lastname)
161+
field: fullname
162+
language: sql
163+
- uses: remove_field
164+
with:
165+
fields:
166+
- field: firstname
167+
- field: lastname
168+
output:
169+
- uses: redis.write
170+
with:
171+
connection: target
172+
data_type: hash
173+
key:
174+
expression: concat(['cust:', customerid])
175+
language: jmespath
176+
```

‎content/integrate/redis-data-integration/data-pipelines/transform-examples/redis-remove-field-example.md

Lines changed: 166 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,166 @@
1+
---
2+
Title: Remove fields from a key
3+
alwaysopen: false
4+
categories:
5+
- docs
6+
- integrate
7+
- rs
8+
- rdi
9+
description: null
10+
group: di
11+
linkTitle: Remove fields
12+
summary: Redis Data Integration keeps Redis in sync with a primary database in near
13+
real time.
14+
type: integration
15+
weight: 40
16+
---
17+
18+
By default, RDI adds fields to
19+
[hash]({{< relref "/develop/data-types/hashes" >}}) or
20+
[JSON]({{< relref "/develop/data-types/json" >}}) objects in the target
21+
database for each of the columns of the source table.
22+
The examples below show how to omit some of those fields from the target data with the
23+
[`remove_field`]({{< relref "/integrate/redis-data-integration/reference/data-transformation/remove_field" >}}) transformation.
24+
25+
## Remove a single field
26+
27+
The first example removes a single field from the data.
28+
The `source` section selects the `employee` table of the
29+
[`chinook`](https://github.com/Redislabs-Solution-Architects/rdi-quickstart-postgres)
30+
database (the optional `db` field here corresponds to the
31+
`sources.<source-name>.connection.database` field defined in
32+
[`config.yaml`]({{< relref "/integrate/redis-data-integration/data-pipelines/data-pipelines#the-configyaml-file" >}})).
33+
34+
In the `transform` section, the `remove_field` transformation removes the
35+
`hiredate` field.
36+
37+
The `output` section specifies `hash` as the `data_type` to write to the target, which
38+
overrides the default setting of `target_data_type` defined in `config.yaml`. Also, the
39+
`output.with.key` section specifies a custom key format of the form `emp:<employeeid>`.
40+
Note that any fields you remove in the `transform` section are not available for
41+
the key calculation in the `output` section.
42+
43+
The full example is shown below:
44+
45+
```yaml
46+
source:
47+
db: chinook
48+
table: employee
49+
transform:
50+
- uses: remove_field
51+
with:
52+
field: hiredate
53+
output:
54+
- uses: redis.write
55+
with:
56+
connection: target
57+
data_type: hash
58+
key:
59+
expression: concat(['emp:', employeeid])
60+
language: jmespath
61+
```
62+
63+
If you queried the generated target data from the default transformation
64+
using [`redis-cli`]({{< relref "/develop/connect/cli" >}}), you would
65+
see something like the following:
66+
67+
```bash
68+
> hgetall emp:8
69+
1) "employeeid"
70+
2) "8"
71+
3) "lastname"
72+
4) "Callahan"
73+
5) "firstname"
74+
6) "Laura"
75+
7) "title"
76+
8) "IT Staff"
77+
9) "reportsto"
78+
10) "6"
79+
11) "birthdate"
80+
12) "-62467200000000"
81+
13) "hiredate"
82+
14) "1078358400000000"
83+
15) "address"
84+
16) "923 7 ST NW"
85+
.
86+
.
87+
```
88+
89+
Using the job file above, the data omits the `hiredate` field:
90+
91+
```bash
92+
> hgetall emp:8
93+
1) "employeeid"
94+
2) "8"
95+
3) "lastname"
96+
4) "Callahan"
97+
5) "firstname"
98+
6) "Laura"
99+
7) "title"
100+
8) "IT Staff"
101+
9) "reportsto"
102+
10) "6"
103+
11) "birthdate"
104+
12) "-62467200000000"
105+
13) "address"
106+
14) "923 7 ST NW"
107+
.
108+
.
109+
```
110+
111+
## Remove multiple fields
112+
113+
The `remove_field` transformation can also remove multiple fields at the same time
114+
if you specify them under a `fields` subsection. The example below is similar
115+
to the previous one but also removes the `birthdate` field:
116+
117+
```yaml
118+
source:
119+
db: chinook
120+
table: employee
121+
transform:
122+
- uses: remove_field
123+
with:
124+
fields:
125+
- field: hiredate
126+
- field: birthdate
127+
output:
128+
- uses: redis.write
129+
with:
130+
connection: target
131+
data_type: hash
132+
key:
133+
expression: concat(['emp:', employeeid])
134+
language: jmespath
135+
```
136+
137+
If you query the data, you can see that it also omits the
138+
`birthdate` field:
139+
140+
```bash
141+
> hgetall emp:8
142+
1) "employeeid"
143+
2) "8"
144+
3) "lastname"
145+
4) "Callahan"
146+
5) "firstname"
147+
6) "Laura"
148+
7) "title"
149+
8) "IT Staff"
150+
9) "reportsto"
151+
10) "6"
152+
11) "address"
153+
12) "923 7 ST NW"
154+
.
155+
.
156+
```
157+
158+
## Using `remove_field` with `add_field`
159+
160+
The `remove_field` transformation is very useful in combination with
161+
[`add_field`]({{< relref "/integrate/redis-data-integration/data-pipelines/transform-examples/redis-add-field-example" >}}).
162+
For example, if you use `add_field` to concatenate a person's first
163+
and last names, you may not need separate `firstname` and `lastname`
164+
fields, so you can use `remove_field` to omit them.
165+
See [Using `add_field` with `remove_field`]({{< relref "/integrate/redis-data-integration/data-pipelines/transform-examples/redis-add-field-example#using-add_field-with-remove_field" >}})
166+
for an example of how to do this.

0 commit comments

Comments
 (0)