You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: content/integrate/redis-data-integration/reference/config-yaml-reference.md
+21-19Lines changed: 21 additions & 19 deletions
Original file line number
Diff line number
Diff line change
@@ -14,7 +14,7 @@ for more information about the role `config.yaml` plays in defining a pipeline.
14
14
15
15
## Note about fully-qualified table names
16
16
17
-
Throughout this document we use the format `<databaseName>.<tableName>` to refer to a fully-qualified table name. This format is actually the one used by MySQL, but for Oracle,
17
+
Throughout this document we use the format `<databaseName>.<tableName>` to refer to a fully-qualified table name. This format is actually the one used by MySQL/MariaDB, but for Oracle,
18
18
SQLServer, and PostgreSQL, you should use `<schemaName>`.`<tableName>` instead.
19
19
20
20
{{< note >}}You can specify the fully-qualified table name `<databaseName>.<tableName>` as
@@ -84,7 +84,7 @@ See the Debezium documentation for more information about the specific connector
84
84
| -- | -- | -- | -- |
85
85
| `host` | `string` | MariaDB, MySQL, Oracle, PostgreSQL, SQLServer| The IP address of the database instance. |
86
86
| `port` | `integer` | MariaDB, MySQL, Oracle, PostgreSQL, SQLServer | The port of the database instance. |
87
-
| `database` | `string` | MariaDB, MySQL, Oracle, PostgreSQL, SQLServer| The name of the database to capture changes from. For `SQL Server` you can define this as comma-separated list of database names. |
87
+
| `database` | `string` | Oracle, PostgreSQL, SQLServer| The name of the database to capture changes from. For `SQL Server` you can define this as comma-separated list of database names. |
88
88
| `database.pdb.name` | `string` | Oracle |The name of the [Oracle Pluggable Database](https://docs.oracle.com/en/database/oracle/oracle-database/19/riwin/about-pluggable-databases-in-oracle-rac.html) that the connector captures changes from. Do not specify this property for a non-CDB installation.<br/> Default: `"ORCLPDB1"`|
89
89
| `database.encrypt` | `string` | SQL Server| If SSL is enabled for your SQL Server database, you should also enable SSL in RDI by setting the value of this property to `true`.<br/> Default: `false`|
90
90
| `database.server.id` | `integer` | MySQL | Numeric ID of this database client, which must be unique across all currently-running database processes in the MySQL cluster.<br/> Default: 1|
@@ -104,15 +104,15 @@ See the Debezium documentation for more information about the specific connector
104
104
| `redis.wait.enabled` | `string` | If Redis is configured with a replica shard, this lets you verify that the data has been written to the replica.<br/> Default: `false`|
105
105
| `redis.wait.timeout.ms` | `integer` | Defines the timeout in milliseconds when waiting for the replica.<br/> Default: `1000`|
106
106
| `redis.wait.retry.enabled` | `string` | Enables retry on wait for replica failure.<br/> Default: `false`|
107
-
| `redis.wait.retry.delay.ms` | `integer` | Defines the delay for retry on wait for replica failure.<br/> Default: `1000`|
107
+
| `redis.wait.retry.delay.ms` | `integer` | Defines the delay (in milliseconds) for retry on wait for replica failure.<br/> Default: `1000`|
108
108
| `redis.retry.initial.delay.ms` | `integer` | Initial retry delay (in milliseconds) when encountering Redis connection or OOM issues. This value will be doubled upon every retry but won’t exceed `redis.retry.max.delay.ms`.<br/> Default: `300`|
109
109
| `redis.retry.max.delay.ms` | `integer` | Maximum delay (in milliseconds) when encountering Redis connection or OOM issues.<br/> Default: `10000`|
110
110
111
111
#### `source`
112
112
113
113
| Name | Type | Source Databases | Description |
114
114
|--|--|--|--|
115
-
| `snapshot.mode` | `string` | MariaDB, MySQL, Oracle, PostgreSQL, SQLServer | Specifies the mode that the connector uses to take snapshots of a captured table.<br/> Default: `"initial"`|
115
+
| `snapshot.mode` | `string` | MariaDB, MySQL, Oracle, PostgreSQL, SQLServer | Specifies the mode that the connector uses to take snapshots of a captured table. See the [Debezium documentation](https://debezium.io/documentation/reference/stable/operations/debezium-server.html) for more details about the available options and configuration.<br/> Default: `"initial"`|
116
116
| `topic.prefix` | `string` | MySQL, Oracle, PostgreSQL, SQLServer| A prefix for all topic names that receive events emitted by this connector.<br/>Default: `"rdi"`|
117
117
| `database.exclude.list` | `string` | MariaDB, MySQL | An optional, comma-separated list of regular expressions that match the names of databases for which you do not want to capture changes. The connector captures changes in any database whose name is not included in `database.exclude.list`. Do not specify the `database` field in the `connection` configuration if you are using the `database.exclude.list` property to filter out databases. |
118
118
| `schema.exclude.list` | `string` | Oracle, PostgreSQL, SQLServer | An optional, comma-separated list of regular expressions that match names of schemas for which you do not want to capture changes. The connector captures changes in any schema whose name is not included in `schema.exclude.list`. Do not specify the `schemas` section if you are using the `schema.exclude.list` property to filter out schemas. |
@@ -124,7 +124,7 @@ See the Debezium documentation for more information about the specific connector
124
124
125
125
### Using custom queries in the initial snapshot {#custom-initial-query}
126
126
127
-
{{< note >}}This section is relevant only for MySQL, Oracle, PostgreSQL, and SQLServer.
127
+
{{< note >}}This section is relevant only for MySQL/MariaDB, Oracle, PostgreSQL, and SQLServer.
128
128
{{< /note >}}
129
129
130
130
By default, the initial snapshot captures all rows from each table.
@@ -136,15 +136,15 @@ After the `snapshot.select.statement.overrides` list, you must then add another
136
136
The format of the property name depends on the database you are using:
137
137
138
138
- For Oracle, SQLServer, and PostrgreSQL, use `snapshot.select.statement.overrides.<SCHEMA_NAME>.<TABLE_NAME>`
139
-
- For MySQL, use: `snapshot.select.statement.overrides<DATABASE_NAME>.<TABLE_NAME>`
139
+
- For MySQL and MariaDB, use: `snapshot.select.statement.overrides<DATABASE_NAME>.<TABLE_NAME>`
140
140
141
141
For example, with PostgreSQL, you would have a configuration like the following:
You must also add the list of columns you want to include in the custom `SELECT` statement using fully-qualified names under "sources.tables". Specify each column in the configuration as shown below:
160
160
161
161
```yaml
162
162
tables:
163
-
schema_name.table_name: # For MySQL: use database_name.table_name
164
-
columns:
165
-
- column_name1 # Each column on a new line
166
-
- column_name2
167
-
- column_name3
163
+
# For MySQL and MariaDB: use database_name.table_name
164
+
schema_name.table_name:
165
+
columns:
166
+
- column_name1 # Each column on a new line
167
+
- column_name2
168
+
- column_name3
168
169
```
169
170
170
171
If you want to capture all columns from a table, you can use empty curly braces `{}` instead of listing all the individual columns:
171
172
172
173
```yaml
173
174
tables:
174
-
schema_name.table_name: {} # Captures all columns
175
+
# Captures all columns. For MySQL and MariaDB: use database_name.table_name.
176
+
schema_name.table_name: {}
175
177
```
176
178
177
179
The example configuration below selects the columns `CustomerId`, `FirstName` and `LastName` from the `customer` table and joins it with the `invoice` table to select customers with total invoices greater than 8000:
178
180
179
181
```yaml
180
182
tables:
181
183
chinook.customer:
182
-
columns:
183
-
- CustomerID
184
-
- FirstName
185
-
- LastName
184
+
columns:
185
+
- CustomerID
186
+
- FirstName
187
+
- LastName
186
188
187
189
advanced:
188
190
source:
@@ -231,7 +233,7 @@ message.
231
233
| `read_batch_size` |`integer`, `string`| Batch size for reading data from the source database.<br/>Default: `2000`<br/>Pattern: `^\${.*}$`<br/>Minimum: `1`|
| `dedup_max_size` |`integer`| Maximum size of the deduplication set.<br/> Default: `1024`<br/>Minimum: `1`<br/>|
236
+
| `dedup_max_size` |`integer`| Maximum number of items in the deduplication set.<br/> Default: `1024`<br/>Minimum: `1`<br/>|
235
237
| `dedup_strategy` |`string`| Deduplication strategy: `reject`- reject messages (dlq), `ignore` \- ignore messages.<br/> (DEPRECATED)<br/>The property `dedup_strategy` is now deprecated. The only supported strategy is `ignore`. Please remove from the configuration.<br/>Default: `"ignore"`<br/>Enum: `"reject"`, `"ignore"`<br/>|
236
238
| `duration` |`integer`, `string`| Time (in ms) after which data will be read from stream even if `read_batch_size` was not reached.<br/> Default: `100`<br/>Pattern: `^\${.*}$`<br/>Minimum: `1`<br/>|
237
239
| `write_batch_size` |`integer`, `string`| The batch size for writing data to target Redis database\. Should be less or equal to `read_batch_size`.<br/> Default: `200`<br/>Pattern: `^\${.*}$`<br/>Minimum: `1`<br/>|
0 commit comments